:W&Bá¾;¥×M c3mvx01yª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0¼çsû‚" 663a8tcffsit c3mvx01yŠ× c3mvx01yjuju11-uc-san-diegoSPAtop"š _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.02917580463791032 epochs‚10  hidden_dim‚128 ' learning_rate‚0.0002266228067151848  num_attention_heads‚8  num_branch_encoder_layers‚3  num_embedding_layers‚10  num_encoder_layers‚6Bjjtttt_vanilla_c3mvx01yb88oquqqwj)jj-spatop-sweepagent-tttt-vanilla-2-spfhcŠ š€åÔÈñ™šš3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64Íë¦7 c3mvx01y&  _runtime‚0  _wandb.runtime‚0v š jÒä -Linux-6.8.0-136-generic-x86_64-with-glibc2.36CPython 3.12.10 ™€åÔè—Ó¢*-of*=/data/spatop/options_files/sweep_tttt_jj/config_c3mvx01y.json*-l* /scratch/logs*-n*jjtttt_vanilla_c3mvx01y2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb)jj-spatop-sweepagent-tttt-vanilla-2-spfhcr/opt/conda/bin/pythonº gjs5h2zsh11h54n5lna9rwhsyc279jk7’MtqÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€€¯‚«"ª€àäÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-1512559d-7724-700d-39a3-dab660c9a52eÊ13.3º gjs5h2zsh11h54n5lna9rwhsyc279jk7µª½Ç2  requirements.txtªµSîW c3mvx01yZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64¿ãË_Y c3mvx01yZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64ðÈ" *  c3mvx01yb trainer/global_step2ŸïÓ4\  c3mvx01yZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64fÞFç/  c3mvx01yb*"trainer/global_step2Á6CÈK  c3mvx01yj: ›€åÔ¸„ˆ´*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ¨wjz“  c3mvx01yj  ›€åÔÐáÀ¡î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 865 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 2.0 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 38.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ÃDá\ c3mvx01yjÊ ›€åÔ¸þ¶¢·Trainable params: 40.9 M Non-trainable params: 86 Total params: 40.9 M Total estimated model params size (MB): 163.524 Modules in train mode: 4413 Modules in eval mode: 0 Total FLOPs: 0 ‚Ø>q c3mvx01yj` ›€åÔð¬Ù¢Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). }‡¡Þ{ c3mvx01yjj ›€åÔÈËá¢Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ¼¼ŠÍy c3mvx01yjh ›€åÔˆÝð¢Xwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). ‹¿îB€ c3mvx01yjo ›€åÔðÃû¢_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). d¡È z c3mvx01yji ›€åÔ𙉣Ywandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). A+t c3mvx01yjc ›€åÔ§£Swandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). Ù®q c3mvx01yj` ›€åÔȯ”£Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ‘Öt c3mvx01yjc ›€åÔÈ ™£Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). »^¤ n c3mvx01yj] ›€åÔÈ‘ž£Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ›® êm c3mvx01yj\ ›€åÔ뢣Lwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). ÚW@db c3mvx01y*Ð  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚0  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"ÎVž†ë c3mvx01yjÙ ›€åÔØÚç§È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ÃïƒN c3mvx01yj¼ ›€åÔ€úô§«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ànê c3mvx01yjØ ž€åÔ€“ª_È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ¬íÒÀO c3mvx01yj½ ž€åÔ¨¾_­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. øߘ‚(:† ª€åÔߟŽ gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚33.0 gpu.0.powerWatts‚154.87' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[|yÂ(:û ª€åÔ€ÝÏ” network.recv‚39949 disk.sda3.out‚ 0.4609375 disk.rbd4.in‚0 disk.sda.in‚0 disk.nvme0c0n1.in‚0 disk.sda.out‚ 0.4609375 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 1.59765625 disk.dm-0.in‚0 disk.nvme0n1.in‚0# memory_percent‚97.6747989654541) proc.memory.percent‚16.96341037750244) disk./.usagePercent‚73.73834571781728 disk.sda3.in‚0 disk.dm-0.out‚ 0.85791015625 disk.dm-1.out‚ 0.4609375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2779.28515625 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.sda1.in‚0" disk.nvme0n1.out‚ 0.85791015625$ disk.nvme0c0n1.out‚ 0.85791015625 disk.sda2.in‚0 cpu‚24.913647528832673 disk.sda2.out‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 380.9609375$ disk./.usageGB‚1098.754638671875 disk.dm-1.in‚0 network.sent‚33319 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚0‡tßì’ ‚(:ˆ ¹€åÔ§ñü gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚105.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©Îÿ´!‚(:ù ¹€åÔ˜«š‘ disk.dm-1.out‚1.21875 disk.nvme0n1.in‚0 disk.sda3.out‚1.21875 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0* proc.memory.percent‚16.963481903076172% disk./.usageGB‚1098.7546005249023 disk.rbd1.out‚0 disk.sda.in‚0 disk.sda.out‚1.21875 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.7421875 disk.sda3.in‚0$ memory_percent‚97.68123626708984) disk./.usagePercent‚73.73834315774192% proc.memory.availableMB‚ 379.90625$ disk.nvme0n1.out‚100.00341796875 disk.nvme0c0n1.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd0.in‚0! disk.dm-0.out‚100.00341796875 network.sent‚55397 network.recv‚66650& disk.nvme0c0n1.out‚100.00341796875 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 2779.296875 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.dm-0.in‚0 cpu‚24.953377217606622—;Äa3"b/ lr-AdamW/pg1"trainer/global_step2:JP<Ñsˆ3#b/ lr-AdamW/pg2"trainer/global_step2:JPK-Jò-$) _wandbruntime‚37  _runtime‚37B%œàÀ%» & lr-AdamW/pg2‚9.184878022368946e-06 trainer/global_step‚49 % _timestamp‚1.7884283530305889e+09 _runtime‚ 37.735401346 _step‚0 & lr-AdamW/pg1‚9.184878022368946e-06l~úYÂ&½ & lr-AdamW/pg1‚9.184878022368946e-06 & lr-AdamW/pg2‚9.184878022368946e-06 trainer/global_step‚49 % _timestamp‚1.7884283530305889e+09 _runtime‚ 37.735401346 _step‚0¥Øßs'bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPYŒâZg(bc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPB]Êe)ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPÀRŸj*bf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JPšh¥xn+bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP:‚°n,bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPô´²Êm-bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPÜ®·sh.bd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP—×i/be BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPãWa0b] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPæ•%üp1bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP”oWól2bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP«Ä¢v3br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP‘7\±s4bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPÜCÚ`5b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPb¶9m6bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP€4n¶h7bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPÅ[»Ln8bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP‹2h9bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPx$$Rn:bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JP.Sèâx;bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPhëæˆmbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP>ôÚÈg?bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP¡n]µo@bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPíÍ"ÒkAbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP›´W{nBbj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPáÞO)iCbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP"€y&qDbm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPãYÊeEba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP=.örkFbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JP£Œ¹ásGbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPML)ÛqHbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP®'Å"gIbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPVzókJbg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPÔËg”rKbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPí£‡glLbh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JP‹cNVWMbS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP¤4“iNbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP=äÊ'jObf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP< 2WiPbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JP‰R«pQbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP ð:ÚmRbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP"@p²hSbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPIÙrTbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP˜MnUbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP¦‚n gVbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPCDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPCDfUkXbg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPçkü fYbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPn·‡;_Zb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPÞ¶ÔDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPRqpupbbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP´Žö{jcbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JP014|rdbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPæÁ{iebe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP $§¡qfbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP7©ï+mgbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP'™oôdhb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JPO̽3libh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP eStjbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPéæ‘*gkbc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP«ëÚRplbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPõéMJnmbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP?ÒÄUonbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP¾Gg‹fobb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPC;rFopbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP § aqb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPn“¿Wfrbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP”3Újsbf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPà4²jntbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPǦ„Anubj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP`nŒÁmvbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP+úÆýswbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP÷Ëàdxb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPÑ> sybo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP" -Ì^zbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPP)wk{bg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JP7¨?Èi|be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPBYÞf}bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPQïlf~bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPüåéìcb_ ÿ®s‰bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPRj[ÉqŠbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP±ïăf‹ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPyB´¸cŒb^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JP5Ô˜lbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPØ;fwŽbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPúÎt.kbf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP.k9Šhbc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP°|0o‘bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPO¯à%m’bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPÕ„Êl“bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPèW}p”bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPU»Éó`•b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP£/c1d–b_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22628 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚645582 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚854 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1629594 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 40735264768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22901 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 43517028352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 866123776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2875281690112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 2834564428288 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚708396 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚231480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 866123776 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954818048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3103168254464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 2834546425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2760889916416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10192158720 _step‚1 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚708398 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚708398 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3146520891904 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 41237680866 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚843 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 181444096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚843 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863047680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚49 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691469 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 2834546425344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 40735264768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚843 BDeviceStñTq’ atsMonitor.on_train_batch_start/active.large_pool.peak‚2605 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚921198 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚231030 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚413 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 41390860288 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2875955288576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2875281690112 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚414100 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3103192194560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 2802144636642 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845965 _runtime‚ 37.736429763 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954818048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10192158720 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚939738 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3146709222912 % _timestamp‚1.7884283530313694e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚413 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 2834564428288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1648136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 866123776.¡B‡'¤O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22901 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 43517028352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 866123776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2875281690112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 2834564428288 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚708396 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚231480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 866123776 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954818048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3103168254464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 2834546425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2760889916416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10192158720 _step‚1 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚708398 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚708398 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3146520891904 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 41237680866 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚843 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 181444096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚843 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863047680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚49 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691469 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 2834546425344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 40735264768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚843 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚921198 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚231030 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚413 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 41390860288 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2875955288576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2875281690112 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚414100 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3103192194560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 2802144636642 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845965 _runtime‚ 37.736429763 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954818048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10192158720 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚939738 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3146709222912 % _timestamp‚1.7884283530313694e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚413 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 2834564428288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1648136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 866123776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚835 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863047680 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚645130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22628 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚645582 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚854 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1629594 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 40735264768òº¿i¥bd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JP«0Bd¦b_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPˆ®Öa¶b\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JPòûUm·bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPÌ@¢¨j¸be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP.®’ n¹bi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPdl»lrºbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP•Rñ}\»bW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPXuÉ f¼ba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP€kÎi½bd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JP–þh¾bc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JP­¡Ïm¿bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPGû‡rÀbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPËí hÁbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPö½õlÂbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP_õ™‰mÃbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP#¨@lÄbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPöQªoÅbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP´"øÅuÆbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPÛ[ælÇbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPÖü£ºkÈbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPÿ¦@ kÉbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPÜÃÃeÊb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP^ÑUœ^ËbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JPK ;eÌb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP¸h– mÍbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPÃL ]ÎbX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPÜWM…jÏbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPÄÕ¬œrÐbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP{ÎÃ]ÑbX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP\ bgÒbb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JP¼r¢ˆqÓbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPºŽºjÔbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPò>…XcÕb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPƒÜciÖbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPž‚¦kf×ba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPÖYjØbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP¿‚2lrÙbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP…èÄnmÚbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPÕ¿`wÛbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP ,áogÜbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPPBÄoÝbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP&0ªhÞbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPn¢0§eßb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JP·Qclgàbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP ¹ mmábh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPŒ{>mâbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP%~30`ãb[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPÍgäbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPý§qoåbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP¢¸É®uæbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPÐ×(bçb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPסE¶pèbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPCº+¢béb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP‰‘—–^êbY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPT—|¬dëb_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPYϸlùbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPó¾bgúbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JP¬´Çhûbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPS¯½kübf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPgÛVýbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPº$™Wqþbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPµµ×jÿbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JP4·Öe€b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP†&Iˆnbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPH±n‚bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPõN‚fƒba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPžµø­w„br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPC°Ûr…bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPÛ š$j†be BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPœo'vo‡bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPµâÙhˆbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JP¬(if‰ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPG5ËnŠbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP5@ i‹bd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JP}Ú!pŒbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPókƒ·nbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPv)äjŽbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP ¾…‚eb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPQµzgbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JP±ÐEŸc‘b^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JPõÙ’k’bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPq$âo“bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP™L¼¢m”bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPû•8`•b[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JP‰Á>p–bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPÀ¸—Df—ba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPÌ< ¼l˜bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP>w=¢d™b_ ¥øqšbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPQi›bd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JP'.Nrœbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP’àçfba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPßzŽmžbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPMó©@iŸbd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JPòùï. ) _wandbruntime‚38  _runtime‚38íV‰¡†M XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2890899464192 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 41399685622 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 44373842432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 181444096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 41554692608 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚722412 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3209216774656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2815794046976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10192158720 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚722412 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 332015104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2890917467136 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚235630 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚49 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2933127768576 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚722414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚958218 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3164818992128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2933127768576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚843 _step‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚658355 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1680632 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚939652 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚413 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 2815811086336 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 866123776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2890899464192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10192158720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚939652 PEDevr7žiceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚422268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2890917467136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚413 $ _timestamp‚1.788428353687174e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚843 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚958218 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 42210301440 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚722414 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3164842932224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 866123776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2857193732598 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 _runtime‚ 38.392400573 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2932454156800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1680632 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1662064 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚657898 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2857865202374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 866123776 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430'¡åÑ&¢ŠM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1680632 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚939652 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚413 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 2815811086336 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 866123776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2890899464192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10192158720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚939652 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚422268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2890917467136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚413 $ _timestamp‚1.788428353687174e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚843 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚958218 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 42210301440 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚722414 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3164842932224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 866123776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2857193732598 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 _runtime‚ 38.392400573 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2932454156800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1680632 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1662064 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚657898 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2857865202374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 866123776 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2890899464192 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 41399685622 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 44373842432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 181444096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 41554692608 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚722412 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3209216774656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2815794046976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10192158720 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚722412 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 332015104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2890917467136 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚235630 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚49 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2933127768576 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚722414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚958218 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3164818992128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2933127768576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚843 _step‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚658355 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424|6Ž‰@£b; loss/FBt1/detection_loss"trainer/global_step2:JPÒ‚vùA¤b< loss/FBt3/assignment_loss"trainer/global_step2:JP9Íû@¥b; loss/FBt2/detection_loss"trainer/global_step2:JPyAb loss/SRqqt2/assignment_loss"trainer/global_step2:JPØdì7¨b2 loss/total_loss"trainer/global_step2:JP]_˜VA©b< loss/FBt1/assignment_loss"trainer/global_step2:JPxÁ$@ªb; loss/FRt1/detection_loss"trainer/global_step2:JPÃ3³±@«b; loss/FRt3/detection_loss"trainer/global_step2:JP–áDÖ@¬b; loss/FBt4/detection_loss"trainer/global_step2:JP<—¹þA­b< loss/FBt4/assignment_loss"trainer/global_step2:JP¥QöB®b= loss/SRqqt3/detection_loss"trainer/global_step2:JP¤DÁC¯b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPAW,@°b; loss/FRt2/detection_loss"trainer/global_step2:JPòB±b= loss/SRqqt2/detection_loss"trainer/global_step2:JPÀº@²b; loss/FBt3/detection_loss"trainer/global_step2:JP‹ô0-³b( epoch"trainer/global_step2:JP«°RÉA´b< loss/FRt2/assignment_loss"trainer/global_step2:JP!ÌCµb> loss/SRqqt4/assignment_loss"trainer/global_step2:JPU23&A¶b< loss/FBt2/assignment_loss"trainer/global_step2:JP-½ Ñ@·b; loss/FRt4/detection_loss"trainer/global_step2:JP9©—äB¸b= loss/SRqqt1/detection_loss"trainer/global_step2:JPCbÖ»B¹b= loss/SRqqt4/detection_loss"trainer/global_step2:JP+£¨Aºb< loss/FRt1/assignment_loss"trainer/global_step2:JP…ôA»b< loss/FRt3/assignment_loss"trainer/global_step2:JPPBTWC¼b> loss/SRqqt3/assignment_loss"trainer/global_step2:JPWƒŽŠ.½) _wandbruntime‚38  _runtime‚38y’«D¾¾ 0loss/FBt3/assignment_loss‚2.7272398273150125 .loss/FBt2/detection_loss‚4.539182234359415 /loss/FRt4/assignment_loss‚7.193120506074694 1loss/SRqqt2/assignment_loss‚5.487948298454285 &loss/total_loss‚118.06226916023843 /loss/FBt1/assignment_loss‚2.683325717323705 .loss/FRt1/detection_loss‚4.362305375245901 /loss/FRt3/detection_loss‚3.1228930159490935 -loss/FBt4/detection_loss‚6.87405077368021 /loss/FBt4/assignment_loss‚2.725699951251348 0loss/SRqqt3/detection_loss‚2.783326800193703 % _timestamp‚1.7884283536890259e+09 1loss/SRqqt1/assignment_loss‚5.508187662091172 .loss/FRt2/detection_loss‚3.470045655965805 0loss/SRqqt2/detection_loss‚2.479062896221876 /loss/FBt3/detection_loss‚7.4271633345633745 epoch‚0 _runtime‚ 38.39400705 _step‚3 .loss/FRt2/assignment_loss‚7.40790301926282 1loss/SRqqt4/assignment_loss‚5.432798835966322 /loss/FBt2/assignment_loss‚2.709683192403693 .loss/FRt4/detection_loss‚4.717936515133058 trainer/global_step‚49 1loss/SRqqt1/detection_loss‚2.7810863585848558 0loss/SRqqt4/detection_loss‚4.895592462271452 /loss/FRt1/assignment_loss‚7.434573797079233 /loss/FRt3/assignment_loss‚7.200430305636659 2loss/SRqqt3/assignment_loss‚5.3301310873868175 .loss/FBt1/detection_loss‚8.768581537823929× ß¼H¿ 1loss/SRqqt2/assignment_loss‚5.487948298454285 &loss/total_loss‚118.06226916023843 /loss/FBt1/assignment_loss‚2.683325717323705 .loss/FRt1/detection_loss‚4.362305375245901 /loss/FRt3/detection_loss‚3.1228930159490935 -loss/FBt4/detection_loss‚6.87405077368021 /loss/FBt4/assignment_loss‚2.725699951251348 0loss/SRqqt3/detection_loss‚2.783326800193703 % _timestamp‚1.7884283536890259e+09 1loss/SRqqt1/assignment_loss‚5.508187662091172 .loss/FRt2/detection_loss‚3.470045655965805 0loss/SRqqt2/detection_loss‚2.479062896221876 /loss/FBt3/detection_loss‚7.4271633345633745 epoch‚0 _runtime‚ 38.39400705 _step‚3 .loss/FRt2/assignment_loss‚7.40790301926282 1loss/SRqqt4/assignment_loss‚5.432798835966322 /loss/FBt2/assignment_loss‚2.709683192403693 .loss/FRt4/detection_loss‚4.717936515133058 trainer/global_step‚49 1loss/SRqqt1/detection_loss‚2.7810863585848558 0loss/SRqqt4/detection_loss‚4.895592462271452 /loss/FRt1/assignment_loss‚7.434573797079233 /loss/FRt3/assignment_loss‚7.200430305636659 2loss/SRqqt3/assignment_loss‚5.3301310873868175 .loss/FBt1/detection_loss‚8.768581537823929 0loss/FBt3/assignment_loss‚2.7272398273150125 .loss/FBt2/detection_loss‚4.539182234359415 /loss/FRt4/assignment_loss‚7.193120506074694ªGN“À‚(:ˆ È€åÔí¡é gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚150.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Š%ÖÁ‚(:ù È€åԚߌ disk.drbd1777.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0$ disk.nvme0c0n1.out‚ 199.888671875* proc.memory.percent‚16.963648796081543) disk./.usagePercent‚73.73832677325974 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.dm-1.in‚0$ memory_percent‚97.69039154052734 proc.cpu.threads‚17 disk.dm-0.out‚ 199.888671875 disk.dm-1.out‚ 1.8046875" disk.nvme0n1.out‚ 199.888671875 cpu‚24.971082804698465 network.recv‚81273 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2779.32421875 network.sent‚71296% disk./.usageGB‚1098.7543563842773 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 1.8046875 disk.sda.out‚ 1.8046875 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.rbd2.out‚3.34375 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚0% proc.memory.availableMB‚ 378.40625 disk.dm-0.in‚0 disk.rbd0.in‚0@é“‚(:ˆ ×€åÔÃéú gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚161.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”Î×Â(:ù ×€åÔÿ½æ) disk./.usagePercent‚73.73832549322206 disk.sda2.in‚0 disk.sda3.out‚ 2.29296875 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚1098.754337310791 disk.sda.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.out‚ 298.9765625 disk.rbd2.out‚ 3.48828125$ memory_percent‚97.69859313964844 disk.dm-1.out‚ 2.29296875 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2779.33203125 network.recv‚113479 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 298.9765625 disk.sda.out‚ 2.29296875 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚0* proc.memory.percent‚16.963696479797363 cpu‚24.987336460036374 proc.cpu.threads‚17 disk.rbd1.out‚0 network.sent‚135504 disk.dm-0.out‚ 298.9765625$ proc.memory.availableMB‚377.0625!vÍç.Ä) _wandbruntime‚72  _runtime‚72Â5XÇÂż ' lr-AdamW/pg1‚1.8557202534990317e-05 ' lr-AdamW/pg2‚1.8557202534990317e-05 trainer/global_step‚99 % _timestamp‚1.7884283878423686e+09 _runtime‚ 72.54710571 _step‚4?£àÆÆÀ ' lr-AdamW/pg1‚1.8557202534990317e-05 ' lr-AdamW/pg2‚1.8557202534990317e-05 trainer/global_step‚99 % _timestamp‚1.7884283878423686e+09 _runtime‚ 72.54710571 _step‚4´Ø.Ç) _wandbruntime‚72  _runtime‚72Ç°c:'ȇO UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 81721070592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5652198367744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚843 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1409196 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚459659 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚863 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚822500 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183540224 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5652216370688 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3274988 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1282161 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5652198367744 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6272075429888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 175748096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 trainer/global_step‚99 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6271899681792 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5506096444416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1847250 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚413 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5734593036800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5734593036800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1409198 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 82073020274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 866123776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚5 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5733919438336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6185705137664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1409198 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 82376666112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 82376666112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _runtime‚ 72.547993753 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 81721070592 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1865790 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1281722 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6185729077760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10192158720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚843 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3256446 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5733919438336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5652216370688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5733919438336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5652216370688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚843 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1409196 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚459659 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚863 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚822500 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183540224 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5652216370688 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3274988 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1282161 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5652198367744 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6272075429888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 175748096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 trainer/global_step‚99 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6271899681792 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5506096444416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1847250 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚413 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5734593036800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5734593036800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1409198 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 82073020274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 866123776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚5 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5733919438336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6185705137664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1409198 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 82376666112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 82376666112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _runtime‚ 72.547993753Û\è.Ê) _wandbruntime‚73  _runtime‚73QØì"—&Ë‘M ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6247355875328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚863 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1423214 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1294420 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 87216459776 _step‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3307484 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5791765516800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5643907069782 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5561017614336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚413 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚99 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5708551406592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1884270 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚464205 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 866123776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚843 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5791765516800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 82235025030 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 866123776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6334403763200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5708569409536 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5791091905024 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 82889455446 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3288916 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5708551406592 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3307484 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 87047887872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1884270 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3288916 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 _runtime‚ 73.213738114 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1423212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183540224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1865704 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 82540498432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0W'¡U›&Ì•M R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5791765516800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 82235025030 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 866123776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6334403763200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5708569409536 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5791091905024 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 82889455446 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3288916 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5708551406592 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3307484 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 87047887872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1884270 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3288916 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 _runtime‚ 73.213738114 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1423212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183540224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1865704 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 82540498432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6247355875328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚863 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1423214 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1294420 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 87216459776 _step‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3307484 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5791765516800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5643907069782 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5561017614336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚413 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚99 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5708551406592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1884270 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚464205 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 866123776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚843½‰èÀ.Í) _wandbruntime‚73  _runtime‚73[ÇŸ¤KÎÅ /loss/FBt1/assignment_loss‚2.572706937789917 0loss/FBt4/assignment_loss‚2.5093316872914633 0loss/SRqqt1/detection_loss‚4.544779650675944 0loss/SRqqt3/detection_loss‚2.863592505455017 .loss/FBt3/detection_loss‚5.859777664829945 epoch‚0 trainer/global_step‚99 2loss/SRqqt1/assignment_loss‚5.4557731946309405 0loss/FBt2/assignment_loss‚2.4901259859402973 /loss/FRt4/assignment_loss‚7.049395912571957 /loss/FRt1/detection_loss‚5.1788312152904625 /loss/FRt2/detection_loss‚2.8103925473988056 .loss/FRt4/detection_loss‚6.980039049137634 /loss/FRt2/assignment_loss‚7.454387877668653 /loss/FRt3/detection_loss‚2.8150103935655557 0loss/SRqqt2/detection_loss‚2.121349616774491 .loss/FBt2/detection_loss‚2.160866697753469 1loss/SRqqt2/assignment_loss‚5.510789019720895 /loss/FBt4/detection_loss‚11.220512797435125 _runtime‚ 73.21470242 0loss/FRt3/assignment_loss‚7.1702529439386335 2loss/SRqqt3/assignment_loss‚5.3079047537686534 1loss/SRqqt4/assignment_loss‚5.428515240550041 /loss/SRqqt4/detection_loss‚5.58709626737982 &loss/total_loss‚128.23522989510136 0loss/FBt3/assignment_loss‚2.5577007079946585 /loss/FBt1/detection_loss‚15.235998996279456 _step‚7 /loss/FRt1/assignment_loss‚7.350098231259515 % _timestamp‚1.7884283885100622e+09SwôéOÏÉ .loss/FBt2/detection_loss‚2.160866697753469 1loss/SRqqt2/assignment_loss‚5.510789019720895 /loss/FBt4/detection_loss‚11.220512797435125 _runtime‚ 73.21470242 0loss/FRt3/assignment_loss‚7.1702529439386335 2loss/SRqqt3/assignment_loss‚5.3079047537686534 1loss/SRqqt4/assignment_loss‚5.428515240550041 /loss/SRqqt4/detection_loss‚5.58709626737982 &loss/total_loss‚128.23522989510136 0loss/FBt3/assignment_loss‚2.5577007079946585 /loss/FBt1/detection_loss‚15.235998996279456 _step‚7 /loss/FRt1/assignment_loss‚7.350098231259515 % _timestamp‚1.7884283885100622e+09 /loss/FBt1/assignment_loss‚2.572706937789917 0loss/FBt4/assignment_loss‚2.5093316872914633 0loss/SRqqt1/detection_loss‚4.544779650675944 0loss/SRqqt3/detection_loss‚2.863592505455017 .loss/FBt3/detection_loss‚5.859777664829945 epoch‚0 trainer/global_step‚99 2loss/SRqqt1/assignment_loss‚5.4557731946309405 0loss/FBt2/assignment_loss‚2.4901259859402973 /loss/FRt4/assignment_loss‚7.049395912571957 /loss/FRt1/detection_loss‚5.1788312152904625 /loss/FRt2/detection_loss‚2.8103925473988056 .loss/FRt4/detection_loss‚6.980039049137634 /loss/FRt2/assignment_loss‚7.454387877668653 /loss/FRt3/detection_loss‚2.8150103935655557 0loss/SRqqt2/detection_loss‚2.121349616774491ˆR©wžÐ‚(:“ æ€åÔ˜æó gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.874' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.374799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃãßÑ‚(:ˆ æ€åÔî­ network.sent‚142103 disk.rbd0.out‚0$ memory_percent‚97.70333766937256 proc.cpu.threads‚17# disk./.usageGB‚1098.75443649292 disk.drbd1777.out‚0 disk.rbd3.in‚0 disw¿¡¯4k.dm-0.in‚0 cpu‚24.994391779314675) disk./.usagePercent‚73.73833214941796 disk.rbd2.out‚ 5.08984375 disk.rbd4.out‚0( proc.memory.availableMB‚ 376.28515625* proc.memory.percent‚16.963791847229004 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 network.recv‚118776 disk.sda3.out‚ 2.94921875 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚398.69091796875 disk.sda2.in‚0! disk.dm-0.out‚398.69091796875# proc.memory.rssMB‚ 2779.34765625 disk.nvme0c0n1.in‚0 disk.rbd4.in‚0 disk.sda.out‚ 2.94921875 disk.sda1.out‚0 disk.rbd1.out‚0& disk.nvme0c0n1.out‚398.69091796875 disk.rbd2.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 2.94921875”Æ"žÒ‚(:“ õ€åÔù• gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.098' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.619600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù¯Ä Ó‚(:€ õ€åÔœÝí disk.sda2.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0! disk.dm-0.out‚399.04248046875 disk.sda1.in‚0% disk./.usageGB‚1098.7551078796387 disk.drbd1777.in‚0 disk.nvme0n1.in‚0 disk.rbd1.out‚0" proc.memory.rssMB‚ 2779.3515625 network.recv‚125604# memory_percent‚97.7025032043457& proc.memory.availableMB‚ 376.421875 network.sent‚192972 disk.sda2.out‚0 disk.rbd2.out‚5.234375 disk.dm-0.in‚0* proc.memory.percent‚16.963815689086914 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.rbd1.in‚0) disk./.usagePercent‚73.73837720674399 disk.rbd2.in‚0 disk.sda3.out‚ 3.5859375 disk.rbd3.in‚0 cpu‚24.99965049760103& disk.nvme0c0n1.out‚399.04248046875 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 3.5859375 disk.dm-1.in‚0 disk.dm-1.out‚ 3.5859375 disk.sda1.out‚0$ disk.nvme0n1.out‚399.04248046875ÂeÃdÔ‚(:… „åÔ‚Òñ gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚93.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øÀç Õ‚(: „åÔÝ“‰ disk.rbd3.out‚0 disk.nvme0c0n1.in‚0 network.recv‚130334) disk./.usagePercent‚73.66710522122656 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 664.494140625 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 664.494140625% disk./.usageGB‚1097.6931037902832 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2779.3515625 cpu‚25.00723924841314 disk.rbd2.out‚ 6.83203125 disk.rbd4.in‚0 disk.dm-0.out‚ 664.494140625 disk.sda1.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 376.1484375* proc.memory.percent‚16.963815689086914 disk.dm-1.out‚ 4.15234375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 4.15234375 disk.rbd1.out‚0 disk.sda2.in‚0$ memory_percent‚97.70417213439941 disk.sda3.out‚ 4.15234375 disk.dm-0.in‚0 disk.sda3.in‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 network.sent‚198999åa0Ö+ _wandbruntime‚107  _runtime‚107Ü@ðóÄ×¾ trainer/global_step‚149 $ _timestamp‚1.788428422676637e+09 _runtime‚ 107.381346639 _step‚8 ' lr-AdamW/pg1‚2.7929527047611693e-05 ' lr-AdamW/pg2‚2.7929527047611693e-05UAôÈØ $ _timestamp‚1.788428422676637e+09 _runtime‚ 107.381346639 _step‚8 ' lr-AdamW/pg1‚2.7929527047611693e-05 ' lr-AdamW/pg2‚2.7929527047611693e-05 trainer/global_step‚1495`3Y0Ù+ _wandbruntime‚107  _runtime‚107é‹¡N™'Ú“O G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚874 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 122253929474 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4883298 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8469868313088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2773302 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1918787 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166488064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8592557186560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 122706876416 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2791842 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2773302 trainer/global_step‚149 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 123362471936 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2109998 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 129289340928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 122706876416 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1918347 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190428160 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2109998 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8593230785024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 _step‚9 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚687447 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9268265960960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8593230785024 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9397555301888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 129122852864 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8373556901890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9397364873728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9268242020864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1230902 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1230900 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 122908359682 % _timestamp‚1.7884284226773272e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2109996 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8592557186560 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 8469850310144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4901840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H'Û—O @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 122908359682 % _timestamp‚1.7884284226773272e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2109996 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8592557186560 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 8469850310144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4901840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2773302 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1918787 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166488064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8592557186560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 122706876416 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2791842 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2773302 trainer/global_step‚149 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 123362471936 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2109998 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 129289340928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 122706876416 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1918347 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190428160 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2109998 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8593230785024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 _step‚9 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚687447 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9268265960960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8593230785024 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9397555301888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 129122852864 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8373556901890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9397364873728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9268242020864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1230902 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1230900 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ÚËÿ0Ü+ _wandbruntime‚108  _runtime‚108™KÚÚÝžM R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8650403265024 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚692342 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172766208 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 123070364438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8526221351936 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4915768 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9460063508992 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8526221351936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8306207102976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4934336 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1931412 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1930972 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8429277467414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 124181913088 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2124014 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2810322 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1239070 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9459866802688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 123526304256 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2124012 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚691904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2791756 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 124181913088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 123526304256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9329892758528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8306224142336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2810322 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8526203348992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚855 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 N8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9329916698624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8429948937190 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4915768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2791756 QEDDE ÊeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1239068 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚874 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196706304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884284233196347e+09 _step‚10 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8526221351936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8306207102976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4934336 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1931412 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1930972 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8429277467414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 124181913088 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2124014 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2810322 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1239070 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9459866802688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 123526304256 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2124012 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚691904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2791756 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 124181913088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 123526304256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9329892758528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8306224142336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2810322 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8526203348992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚855 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 N8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9329916698624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8429948937190 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4915768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2791756 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1239068 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚874 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196706304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884284233196347e+09 _step‚10 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12522732546048 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 164348277760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11451194934784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3699354 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2555783 _step‚13 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1639302 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2555331 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚916481 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6528692 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10996526539776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3717894 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 163089268882 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2810796 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 171953641984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12522920877056 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 172118032896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884284573016715e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3699354 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 163743699090 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚855 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 164348277760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11159598769298 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚874 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11451194934784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11287502252544 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 142.006631031 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1639300 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2810796 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11287520255488 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11451868533248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2810798 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3717894 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12350802844160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2810798 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6510150 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11451868533248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10996509500416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 163692682240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 142.006631031 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1639300 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2810796 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11287520255488 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11451868533248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2810798 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3717894 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12350802844160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2810798 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6510150 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11451868533248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10996509500416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 163692682240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 HDèÅ«4eviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12522732546048 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 164348277760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11451194934784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3699354 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2555783 _step‚13 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1639302 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2555331 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚916481 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6528692 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10996526539776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3717894 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 163089268882 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2810796 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 171953641984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12522920877056 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 172118032896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884284573016715e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3699354 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 163743699090 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚855 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 164348277760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11159598769298 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚874 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11451194934784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ÒìÐI0ì+ _wandbruntime‚142  _runtime‚142(#«´&í®M J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2568073 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3717808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11508367401472 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11051430670336 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11215319334822 _step‚14 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2824814 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚855 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 172805564416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚874 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12585431912448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1647470 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11509041013248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1647468 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6561188 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2824814 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚920605 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6542620 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11215990804598 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884284579609582e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2824812 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚921063 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 164560134262 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11343855291392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 142.666017344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11343855291392 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844¢~2Þ¸&î²M VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11509041013248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1647468 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6561188 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2824814 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚920605 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6542620 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11215990804598 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884284579609582e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2824812 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚921063 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 164560134262 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11343855291392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 142.666017344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11343855291392 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2568073 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3717808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11508367401472 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11051430670336 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11215319334822 _step‚14 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2824814 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚855 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 172805564416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚874 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12585431912448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1647470 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414}ñ[ò0ï+ _wandbruntime‚142  _runtime‚142C¡˜JðÄ 0loss/FRt1/assignment_loss‚7.3352493110455965 0loss/SRqqt2/assignment_loss‚4.51279081995525 1loss/SRqqt3/assignment_loss‚4.134547273929303 0loss/SRqqt4/assignment_loss‚4.47805912759569 /loss/FBt3/assignment_loss‚0.658311670502791 /loss/FRt3/detection_loss‚2.6599316345182835 .loss/FRt4/detection_loss‚4.541303187608719 0loss/SRqqt4/detection_loss‚3.759471763504876 0loss/FBt4/assignment_loss‚0.7095427552858988 trainer/global_step‚199 $ _timestamp‚1.788428457962461e+09 _runtime‚ 142.667064777 _step‚15 0loss/SRqqt1/detection_loss‚3.963630165471587 /loss/FBt4/detection_loss‚11.051708694299062 /loss/FRt3/assignment_loss‚7.112290550161291 .loss/FRt4/assignment_loss‚7.34917672475179 0loss/FBt1/assignment_loss‚0.6960741889973482 .loss/FBt3/detection_loss‚6.573492563687838 &loss/total_loss‚121.68207032832392 /loss/FRt2/detection_loss‚2.3161752129142936 0loss/SRqqt2/detection_loss‚1.839931886725401 /loss/FBt2/detection_loss‚1.3820709723780449 .loss/FBt1/detection_loss‚27.43541956941287 /loss/FRt2/assignment_loss‚7.404284094319199 1loss/SRqqt1/assignment_loss‚4.503614442293034 .loss/FRt1/detection_loss‚4.624063668282409 1loss/FBt2/assignment_loss‚0.43090859567051504 0loss/SRqqt3/detection_loss‚2.210021455012835 epoch‚0t.¦ NñÈ 0loss/SRqqt3/detection_loss‚2.210021455012835 epoch‚0 0loss/FRt1/assignment_loss‚7.3352493110455965 0loss/SRqqt2/assignment_loss‚4.51279081995525 1loss/SRqqt3/assignment_loss‚4.134547273929303 0loss/SRqqt4/assignment_loss‚4.47805912759569 /loss/FBt3/assignment_loss‚0.658311670502791 /loss/FRt3/detection_loss‚2.6599316345182835 .loss/FRt4/detection_loss‚4.541303187608719 0loss/SRqqt4/detection_loss‚3.759471763504876 0loss/FBt4/assignment_loss‚0.7095427552858988 trainer/global_step‚199 $ _timestamp‚1.788428457962461e+09 _runtime‚ 142.667064777 _step‚15 0loss/SRqqt1/detection_loss‚3.963630165471587 /loss/FBt4/detection_loss‚11.051708694299062 /loss/FRt3/assignment_loss‚7.112290550161291 .loss/FRt4/assignment_loss‚7.34917672475179 0loss/FBt1/assignment_loss‚0.6960741889973482 .loss/FBt3/detection_loss‚6.573492563687838 &loss/total_loss‚121.68207032832392 /loss/FRt2/detection_loss‚2.3161752129142936 0loss/SRqqt2/detection_loss‚1.839931886725401 /loss/FBt2/detection_loss‚1.3820709723780449 .loss/FBt1/detection_loss‚27.43541956941287 /loss/FRt2/assignment_loss‚7.404284094319199 1loss/SRqqt1/assignment_loss‚4.503614442293034 .loss/FRt1/detection_loss‚4.624063668282409 1loss/FBt2/assignment_loss‚0.43090859567051504Æ“,Ržò‚(:“ ±åÔáÎØ gpu.0.gpu‚27.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚153.264' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.652800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J¨†Æó‚(:„ ±åÔó÷Õ‘ cpu‚25.01486441085728 disk.rbd4.in‚0 network.sent‚260863 disk.dm-1.out‚6.0625 disk.rbd1.in‚0 disk.rbd4.out‚0 disk.sda.out‚6.0625$ memory_percent‚97.70586490631104 disk.nvme0n1.in‚0 disk.nvme0c0n1.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0# proc.memory.rssMB‚ 2779.40234375% disk./.usageGB‚1097.6930923461914 disk.rbd2.out‚ 10.296875 disk.sda1.in‚0& disk.nvme0c0n1.out‚666.38427734375 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.out‚6.0625 disk.rbd3.out‚0 disk.sda.in‚0$ disk.nvme0n1.out‚666.38427734375 disk.rbd1.out‚ 0.0859375 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 network.recv‚144942) disk./.usagePercent‚73.66710445320396 disk.sda3.in‚0 disk.drbd1777.in‚0! disk.dm-0.out‚666.38427734375* proc.memory.percent‚16.964125633239746 disk.dm-0.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 375.87109375-”¬‰’ô‚(:‡ ÀåÔŽˆÖ gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.391' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.4782 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛŠ[/õ‚(:‡ ÀåÔ‚ú network.sent‚310519 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2779.40234375 disk.sda.out‚ 6.90234375 disk.rbd0.in‚0 disk.rbd2.out‚ 10.3203125 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.nvme0n1.in‚0( proc.memory.availableMB‚ 376.07421875 cpu‚25.01808911617185% disk./.usageGB‚1097.6931228637695 disk.sda3.in‚0 disk.sda3.out‚ 6.90234375 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.dm-0.out‚ 667.20703125 disk.dm-1.out‚ 6.90234375 network.recv‚150753 disk.rbd1.out‚ 0.0859375! disk.nvme0n1.out‚ 667.20703125* proc.memory.percent‚16.964125633239746) disk./.usagePercent‚73.66710650126423 disk.rbd1.in‚0 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.out‚ 667.20703125 disk.rbd3.in‚0# memory_percent‚97.7046251296997UÓ¬-0ö+ _wandbruntime‚176  _runtime‚176­¾‡}Ä÷¾ & lr-AdamW/pg2‚4.667417607285444e-05 trainer/global_step‚249 % _timestamp‚1.7884284920057116e+09 _runtime‚ 176.710437014 _step‚16 & lr-AdamW/pg1‚4.667417607285444e-05f5:ZÈø & lr-AdamW/pg2‚4.667417607285444e-05 trainer/global_step‚249 % _timestamp‚1.7884284920057116e+09 _runtime‚ 176.710437014 _step‚16 & lr-AdamW/pg1‚4.667417607285444e-05…²˜0ù+ _wandbruntime‚176  _runtime‚176…çß‹»ú¦O QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1146382 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 162293760 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14309832683008 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4643946 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversiz°à¸´ñ"e_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4625406 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14105154194944 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8155544 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3511598 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3194523 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14310506281472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _runtime‚ 176.711351838 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚860 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2047702 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2047700 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 205334083584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 203924608290 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186233856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8137002 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15648159130624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13946312106274 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13945640636706 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14310506281472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3194082 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15433315787264 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 214843343360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884284920064108e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 204678488064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚879 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13741733067776 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3511598 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520ÀuG›°'ûªO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1146382 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 162293760 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14309832683008 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4643946 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4625406 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14105154194944 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8155544 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3511598 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3194523 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14310506281472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _runtime‚ 176.711351838 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚860 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2047702 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2047700 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 205334083584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 203924608290 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186233856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8137002 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15648159130624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13946312106274 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13945640636706 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14310506281472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3194082 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15433315787264 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 214843343360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884284920064108e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 204678488064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚879 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13741733067776 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3511598 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464û"‘¢0ü+ _wandbruntime‚177  _runtime‚177NÊ$7¶&ý°M QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3525612 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13796637198336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4662426 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15710660066304 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14002032672006 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3525614 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1151422 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4643860 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8188040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2055868 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 204741043254 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 205497915904 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚18 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 206153524736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8169472 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3207292 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2055870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884284926577017e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3525612 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8169472 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1150983 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 205395473670 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14161525236736 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15710852578304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15494966524928 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8188040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3206851 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 205497915904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4643860 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14367005149696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15494990465024 _runtime‚ 177.363027813 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 215862113280 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0N<ÏVþ´M J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15710660066304 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14002032672006 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3525614 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1151422 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4643860 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8188040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2055868 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 204741043254 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 205497915904 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚18 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 206153524736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8169472 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3207292 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2055870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884284926577017e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3525612 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8169472 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1150983 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 205395473670 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/activÊ.de.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14161525236736 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15710852578304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15494966524928 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8188040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3206851 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 205497915904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4643860 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14367005149696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15494990465024 _runtime‚ 177.363027813 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 215862113280 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3525612 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13796637198336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4662426T?0ÿ+ _wandbruntime‚177  _runtime‚177®o¡qO€É .loss/FRt1/detection_loss‚4.128681598640069 .loss/FBt3/detection_loss‚6.279010034821652 2loss/SRqqt4/assignment_loss‚3.6987666808641873 0loss/FBt1/assignment_loss‚0.4734639460220933 1loss/FBt3/assignment_loss‚0.34474582257853065 0loss/FBt4/assignment_loss‚0.6333373032291146 1loss/SRqqt3/detection_loss‚2.2044270203914493 0loss/SRqqt4/detection_loss‚3.325665969115037 /loss/FRt3/assignment_loss‚7.102260200344786 -loss/FRt4/detection_loss‚4.90577573246426 .loss/FBt1/detection_loss‚10.23896598815918 /loss/FRt1/assignment_loss‚7.453704833984375 1loss/FBt2/assignment_loss‚0.44255677979201713 .loss/FRt3/detection_loss‚2.820993108218726 trainer/global_step‚249 1loss/SRqqt1/assignment_loss‚3.708324149597523 2loss/SRqqt3/assignment_loss‚3.2359995618462563 /loss/FRt2/detection_loss‚2.9326633972426257 1loss/SRqqt1/detection_loss‚3.9431991362294485 .loss/FBt4/detection_loss‚9.496728393961401 # _timestamp‚1.78842849265937e+09 _runtime‚ 177.363989425 %loss/total_loss‚98.98574696825831 epoch‚0 0loss/FRt2/assignment_loss‚7.4584334779668735 /loss/FRt4/assignment_loss‚7.075618849860297 1loss/SRqqt2/detection_loss‚1.5682749548644024 /loss/FBt2/detection_loss‚1.4784185840719755 _step‚19 1loss/SRqqt2/assignment_loss‚4.035731443992028ÏîÆSÍ .loss/FRt1/detection_loss‚4.128681598640069 .loss/FBt3/detection_loss‚6.279010034821652 2loss/SRqqt4/assignment_loss‚3.6987666808641873 0loss/FBt1/assignment_loss‚0.4734639460220933 1loss/FBt3/assignment_loss‚0.34474582257853065 0loss/FBt4/assignment_loss‚0.6333373032291146 1loss/SRqqt3/detection_loss‚2.2044270203914493 0loss/SRqqt4/detection_loss‚3.325665969115037 /loss/FRt3/assignment_loss‚7.102260200344786 -loss/FRt4/detection_loss‚4.90577573246426 .loss/FBt1/detection_loss‚10.23896598815918 /loss/FRt1/assignment_loss‚7.453704833984375 1loss/FBt2/assignment_loss‚0.44255677979201713 .loss/FRt3/detection_loss‚2.820993108218726 trainer/global_step‚249 1loss/SRqqt1/assignment_loss‚3.708324149597523 2loss/SRqqt3/assignment_loss‚3.2359995618462563 /loss/FRt2/detection_loss‚2.9326633972426257 1loss/SRqqt1/detection_loss‚3.9431991362294485 .loss/FBt4/detection_loss‚9.496728393961401 # _timestamp‚1.78842849265937e+09 _runtime‚ 177.363989425 %loss/total_loss‚98.98574696825831 epoch‚0 0loss/FRt2/assignment_loss‚7.4584334779668735 /loss/FRt4/assignment_loss‚7.075618849860297 1loss/SRqqt2/detection_loss‚1.5682749548644024 /loss/FBt2/detection_loss‚1.4784185840719755 _step‚19 1loss/SRqqt2/assignment_loss‚4.035731443992028ÛWlAœ‚‚(:‘ ÏåÔÚî° gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.7' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.339999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²¿¿”ƒ‚(:û ÏåÔÀ©° disk.dm-0.out‚ 667.515625 disk.dm-1.out‚7.640625! disk.nvme0c0n1.out‚ 667.515625 network.recv‚155931 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 375.7265625 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.nvme0n1.out‚ 667.515625% disk./.usageGB‚1097.6931228637695 disk.rbd2.in‚0 disk.sda.out‚7.640625 disk.sda1.in‚0 disk.sda3.out‚7.640625 disk.rbd3.out‚0 disk.rbd4.out‚0$ memory_percent‚97.70674705505371" proc.memory.rssMB‚ 2779.4140625) disk./.usagePercent‚73.66710650126423 disk.sda2.in‚0 disk.rbd4.in‚0 disk.sda.in‚0 cpu‚25.019090359661867 proc.cpu.threads‚17 network.sent‚317354 disk.rbd1.out‚ 0.1484375 disk.nvme0c0n1.in‚0* proc.memory.percent‚16.964197158813477 disk.sda2.out‚0 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 12.0390625 disk.drbd1777.in‚0 disk.drbd1777.out‚0€ª‘mž„‚(:“ ÞåÔ€©þ gpu.0.gpu‚43.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.948' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.789600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ A…‚(:„ ÞåÔŠ¾Œ’ disk.nvme0n1.in‚0( proc.memory.availableMB‚ 375.34765625 disk.rbd3.in‚0# disk.nvme0n1.out‚668.0380859375 proc.cpu.threads‚17 disk.rbd1.out‚ 0.1484375 disk.nvme0c0n1.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0* proc.memory.percent‚16.964244842529297 network.sent‚366724 disk.rbd4.in‚0" memory_percent‚97.709059715271! proc.memory.rssMB‚ 2779.421875) disk./.usagePercent‚73.66711187742244$ disk./.usageGB‚1097.693202972412 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 12.12890625 disk.dm-0.out‚668.0380859375 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.sda.out‚8.234375% disk.nvme0c0n1.out‚668.0380859375 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-1.out‚8.234375 cpu‚25.02191522391347 network.recv‚161871 disk.rbd2.in‚0 disk.sda3.out‚8.234375 disk.rbd4.out‚0Zl±†‚(:’ íåÔàƒ‰ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚88.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.750200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀÿU÷‡‚(:“ íåÔÃ’¶ disk.rbd3.out‚0( proc.memory.availableMB‚ 375.58984375# proc.memory.rssMB‚ 2779.42578125 proc.cpu.threads‚17) disk./.usagePercent‚73.65563557167752 disk.rbd1.out‚ 0.1806640625 disk.rbd2.out‚ 13.78515625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 network.sent‚373662 disk.dm-0.in‚0 disk.sda2.out‚0% disk.nvme0c0n1.out‚670.1943359375 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.dm-1.out‚ 8.92578125 disk.nvme0n1.in‚0 network.recv‚167025 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 8.92578125 disk.sda.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0* proc.memory.percent‚16.964268684387207% disk./.usageGB‚1097.5221977233887 disk.dm-0.out‚670.1943359375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 cpu‚25.023009936065083 disk.sda.out‚ 8.92578125# disk.nvme0n1.out‚670.1943359375 disk.nvme0c0n1.in‚0 disk.rbd4.in‚0$ memory_percent‚97.70758152008057e«›0ˆ+ _wandbruntime‚211  _runtime‚211x&xĉ¾ trainer/global_step‚299 % _timestamp‚1.7884285267600164e+09 _runtime‚ 211.464803589 _step‚20 & lr-AdamW/pg1‚5.604650058547581e-05 & lr-AdamW/pg2‚5.604650058547581e-05vÎìŒÈŠÂ & lr-AdamW/pg1‚5.604650058547581e-05 & lr-AdamW/pg2‚5.604650058547581e-05 trainer/global_step‚299 % _timestamp‚1.7884285267600164e+09 _runtime‚ 211.464803589 _step‚20¶x 0‹+ _wandbruntime‚211  _runtime‚211É{ Á«'Œ¥O Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17168470431232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18773521393664 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16731682504114 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4212398 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16922806137344 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16732353973682 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17168470431232 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9763854 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16922806137344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17169144029696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16922824140288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 245664293888 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4212396 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚299 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5551458 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚21 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 257668723200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5569998 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456100 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16486922556416 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3832986 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16486939595776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 _runtime‚ 211.465591433 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 246319889408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4212396 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚890 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 244759947698 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3833431 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 245414377906 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4212398 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1376886 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18515876610560 B¹!©O D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 245664293888 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4212396 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚299 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5551458 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚21 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 257668723200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5569998 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456100 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16486922556416 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3832986 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16486939595776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 _runtime‚ 211.465591433 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 246319889408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4212396 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚890 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 244759947698 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3833431 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 245414377906 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4212398 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1376886 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18515876610560 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17168470431232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18773521393664 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16731682504114 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4212398 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16922806137344 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16732353973682 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17168470431232 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9763854 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16922806137344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17169144029696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16922824140288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocaten™ˆœöd‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 258527903744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 247139330560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18836031311872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9814892 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5588478 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18836219629568 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18577527348224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16979177179136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18577503408128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16979159176192 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16979177179136 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 247139330560 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5588478 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5569912 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16788074539414 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16787403069638 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9814892 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3845733 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4226414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 245576382662 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9796324 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4226412 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1381911 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 188317696 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 258692281344 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17226316509696 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 $ _timestamp‚1.788428527425862e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5569912 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚871 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16541843726336 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16541826686976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 246483721728 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3846181 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 164377600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚22 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9796324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 246230813078 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16979159176192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2464268 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚299 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2464270 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844Ç0‚Û¹&³M FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16979177179136 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 247139330560 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5588478 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5569912 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16788074539414 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16787403069638 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9814892 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3845733 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4226414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 245576382662 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9796324 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4226412 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1381911 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 188317696 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 258692281344 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17226316509696 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 $ _timestamp‚1.788428527425862e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5569912 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚871 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16541843726336 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16541826686976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 246483721728 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3846181 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 164377600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚22 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9796324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 246230813078 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16979159176192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2464268 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚299 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2464270 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚ 212.131025051 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17226316509696 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17225642897920 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4226414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 258527903744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 247139330560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18836031311872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9814892 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5588478 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18836219629568 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18577527348224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16979177179136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18577503408128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16979159176192ÐG‚0‘+ _wandbruntime‚212  _runtime‚212ŠÇhçQ’Ë 0loss/SRqqt4/detection_loss‚4.111726679262661 /loss/FRt1/assignment_loss‚7.473846090600846 /loss/FRt2/detection_loss‚3.0672618676469012 /loss/FRt3/detection_loss‚2.4808242470026016 /loss/FRt2/assignment_loss‚7.383886166338651 1loss/SRqqt4/assignment_loss‚3.497670258794512 1loss/SRqqt2/detection_loss‚1.3413339500073125 &loss/total_loss‚103.60849861055343 0loss/FBt1/assignment_loss‚0.2872777567245066 trainer/global_step‚299 1loss/SRqqt2/assignment_loss‚3.756912314891815 /loss/FBt1/detection_loss‚16.297723975777625 .loss/FBt3/detection_loss‚2.763712753340894 % _timestamp‚1.7884285274272938e+09 _step‚23 /loss/FRt3/assignment_loss‚7.262926705181599 0loss/FBt4/assignment_loss‚0.7757086766068824 .loss/FBt2/detection_loss‚2.229679215114395 _runtime‚ 212.131904558 -loss/FRt4/detection_loss‚6.97332152023035 /loss/FBt4/detection_loss‚10.115520879626274 epoch‚0 0loss/FBt3/assignment_loss‚0.3795837736049411 .loss/FRt1/detection_loss‚3.633720465162967 1loss/SRqqt1/detection_loss‚3.4688137126681595 /loss/FRt4/assignment_loss‚7.259631437413833 2loss/SRqqt1/assignment_loss‚3.5227377259212993 2loss/SRqqt3/assignment_loss‚3.3214973496719145 0loss/FBt2/assignment_loss‚0.6424076169330094 1loss/SRqqt3/detection_loss‚1.5607734720294797Í]U“Ï _runtime‚ 212.131904558 -loss/FRt4/detection_loss‚6.97332152023035 /loss/FBt4/detection_loss‚10.115520879626274 epoch‚0 0loss/FBt3/assignment_loss‚0.3795837736049411 .loss/FRt1/detection_loss‚3.633720465162967 1loss/SRqqt1/detection_loss‚3.4688137126681595 /loss/FRt4/assignment_loss‚7.259631437413833 2loss/SRqqt1/assignment_loss‚3.5227377259212993 2loss/SRqqt3/assignment_loss‚3.3214973496719145 0loss/FBt2/assignment_loss‚0.6424076169330094 1loss/SRqqt3/detection_loss‚1.5607734720294797 0loss/SRqqt4/detection_loss‚4.111726679262661 /loss/FRt1/assignment_loss‚7.473846090600846 /loss/FRt2/detection_loss‚3.0672618676469012 /loss/FRt3/detection_loss‚2.4808242470026016 /loss/FRt2/assignment_loss‚7.383886166338651 1loss/SRqqt4/assignment_loss‚3.497670258794512 1loss/SRqqt2/detection_loss‚1.3413339500073125 &loss/total_loss‚103.60849861055343 0loss/FBt1/assignment_loss‚0.2872777567245066 trainer/global_step‚299 1loss/SRqqt2/assignment_loss‚3.756912314891815 /loss/FBt1/detection_loss‚16.297723975777625 .loss/FBt3/detection_loss‚2.763712753340894 % _timestamp‚1.7884285274272938e+09 _step‚23 /loss/FRt3/assignment_loss‚7.262926705181599 0loss/FBt4/assignment_loss‚0.7757086766068824 .loss/FBt2/detection_loss‚2.229679215114395¬O“”‚(:ˆ üåÔÑÜî gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-žŒ“•‚(:Ž üåÔ—¸‘ disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 network.sent‚379627 disk.rbd2.out‚ 15.3828125 disk.dm-1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.65565682030287 disk.sda3.out‚9.390625 disk.rbd3.out‚0 disk.sda.in‚0$ disk.nvme0n1.out‚670.95361328125 disk.rbd0.out‚0( proc.memory.availableMB‚ 375.51953125# proc.memory.rssMB‚ 2779.42578125* proc.memory.percent‚16.964268684387207 cpu‚25.02448141909589 network.recv‚171589 disk.rbd4.out‚0 disk.dm-0.in‚0 disk.sda.out‚9.390625 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 0.1806640625 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚670.95361328125$ memory_percent‚97.70801067352295% disk./.usageGB‚1097.5225143432617 disk.drbd1777.out‚0 disk.rbd4.in‚0! disk.dm-0.out‚670.95361328125 disk.dm-1.out‚9.390625 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.rbd1.in‚0‰œˆ£’–‚(:‡ ‹‚åÔÄÏû gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚89.289' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊR®Ë#—‚(:˜ ‹‚åÔÊÇŸ‘ network.recv‚177610 disk.dm-1.in‚0$ disk.nvme0n1.out‚804.41845703125 disk.rbd0.in‚0 disk.sda.out‚ 10.02734375 disk.rbd2.out‚ 15.52734375% disk./.usageGB‚1097.6525230407715 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 10.02734375 disk.rbd2.in‚0 disk.dm-0.in‚0 network.sent‚429910) disk./.usagePercent‚73.66438181307731 disk.rbd1.out‚ 0.24365234375 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.rbd4.out‚0" proc.memory.rssMB‚ 2779.4296875& disk.nvme0c0n1.out‚804.41845703125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 10.02734375 disk.drbd1777.out‚0 disk.sda1.out‚0* proc.memory.percent‚16.964292526245117 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0! disk.dm-0.out‚804.41845703125 disk.sda.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 375.26953125 disk.rbd1.in‚0 disk.rbd4.in‚0# memory_percent‚97.7095365524292 cpu‚25.024954078739515¾ö)80˜+ _wandbruntime‚246  _runtime‚246G„ìvÄ™¾ trainer/global_step‚349 % _timestamp‚1.7884285615378761e+09 _runtime‚ 246.242615003 _step‚24 & lr-AdamW/pg1‚6.541882509809719e-05 & lr-AdamW/pg2‚6.541882509809719e-05Ù öȚ trainer/global_step‚349 % _timestamp‚1.7884285615378761e+09 _runtime‚ 246.242615003 _step‚24 & lr-AdamW/pg1‚6.541882509809719e-05 & lr-AdamW/pg2‚6.541882509809719e-05뢃0›+ _wandbruntime‚246  _runtime‚246ŧtoÙœªO L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4472069 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6477510 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4913196 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19740458079744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19517724371522 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2864502 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 286650099712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19740476082688 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1608040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19518395841090 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚871 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚349 _step‚25 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 300820244480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚ 246.243545496 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6496050 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6496050 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1607569 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4913198 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2864500 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20027108179456 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884285615386207e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19517724371522 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2864502 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 286650099712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19740476082688 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1608040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19518395841090 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚871 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚349 _step‚25 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 300820244480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚ 246.243545496 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6496050 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6496050 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1607569 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4913198 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2864500 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20027108179456 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884285615386207e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4484810 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1612142 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19574116406822 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20084954257920 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 286411722070 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21660040291328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 287066152486 _runtime‚ 246.894445137 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2872670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6495964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4927212 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11423176 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11441744 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 287469527552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 301508251136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4927214 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4927212 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 288125136384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19796829121536 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20084954257920 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6514530 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19573444937046 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19796811118592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4485275 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21961738957312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21961548542464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11441744 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1612605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11423176 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 288125136384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414M½lýh ¸M REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4927214 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4927212 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 288125136384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19796829121536 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20084954257920 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6514530 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19573444937046 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19796811118592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4485275 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21961738957312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21961548542464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11441744 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1612605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11423176 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 288125136384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6514530 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20084280646144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚871 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19287033214976 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4927214 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884285621892836e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4484810 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1612142 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19574116406822 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20084954257920 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 286411722070 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21660040291328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 287066152486 _runtime‚ 246.894445137 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2872670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6495964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4927212 TFDevicoÝgqV eStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11423176 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11441744 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 287469527552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 301508251136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414NQÏ0¡+ _wandbruntime‚246  _runtime‚246«ËòQ¢Ë 1loss/SRqqt2/assignment_loss‚3.625836947025397 0loss/FBt4/assignment_loss‚0.5842718700878322 .loss/FRt2/detection_loss‚4.468751667783811 /loss/FRt3/detection_loss‚2.6114821017340377 .loss/FRt4/detection_loss‚4.275629537446158 _step‚27 1loss/SRqqt3/detection_loss‚2.1457903231888755 /loss/FRt2/assignment_loss‚7.410304546356201 2loss/SRqqt3/assignment_loss‚3.3463814137345653 .loss/FRt1/detection_loss‚2.942093434157195 0loss/SRqqt1/detection_loss‚2.297704108170609 /loss/FBt4/detection_loss‚12.401894051295061 % _timestamp‚1.7884285621909187e+09 0loss/FBt2/assignment_loss‚0.6768597121040026 .loss/FBt1/detection_loss‚17.95967153377003 .loss/FBt3/detection_loss‚3.225433900105682 /loss/FRt1/assignment_loss‚7.361841572655572 2loss/SRqqt1/assignment_loss‚2.4866818225205836 2loss/SRqqt4/assignment_loss‚3.2863984211631445 0loss/FBt1/assignment_loss‚0.5080265383650031 &loss/total_loss‚104.13259594728473 trainer/global_step‚349 /loss/FRt4/assignment_loss‚7.160298574538458 0loss/SRqqt4/detection_loss‚3.816538322231044 /loss/FRt3/assignment_loss‚7.220310228842276 1loss/FBt3/assignment_loss‚0.23030202313252215 0loss/SRqqt2/detection_loss‚2.583184935217355 /loss/FBt2/detection_loss‚1.5069083616593137 epoch‚0 _runtime‚ 246.895549778~IãU£Ï .loss/FRt1/detection_loss‚2.942093434157195 0loss/SRqqt1/detection_loss‚2.297704108170609 /loss/FBt4/detection_loss‚12.401894051295061 % _timestamp‚1.7884285621909187e+09 0loss/FBt2/assignment_loss‚0.6768597121040026 .loss/FBt1/detection_loss‚17.95967153377003 .loss/FBt3/detection_loss‚3.225433900105682 /loss/FRt1/assignment_loss‚7.361841572655572 2loss/SRqqt1/assignment_loss‚2.4866818225205836 2loss/SRqqt4/assignment_loss‚3.2863984211631445 0loss/FBt1/assignment_loss‚0.5080265383650031 &loss/total_loss‚104.13259594728473 trainer/global_step‚349 /loss/FRt4/assignment_loss‚7.160298574538458 0loss/SRqqt4/detection_loss‚3.816538322231044 /loss/FRt3/assignment_loss‚7.220310228842276 1loss/FBt3/assignment_loss‚0.23030202313252215 0loss/SRqqt2/detection_loss‚2.583184935217355 /loss/FBt2/detection_loss‚1.5069083616593137 epoch‚0 _runtime‚ 246.895549778 1loss/SRqqt2/assignment_loss‚3.625836947025397 0loss/FBt4/assignment_loss‚0.5842718700878322 .loss/FRt2/detection_loss‚4.468751667783811 /loss/FRt3/detection_loss‚2.6114821017340377 .loss/FRt4/detection_loss‚4.275629537446158 _step‚27 1loss/SRqqt3/detection_loss‚2.1457903231888755 /loss/FRt2/assignment_loss‚7.410304546356201 2loss/SRqqt3/assignment_loss‚3.3463814137345653gVŠª“¤‚(:ˆ š‚åÔ¹€ê gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚135.403' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.0806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&TÀ襂(: š‚åÔµ¸¯ disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.nvme0n1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.out‚ 10.609375 disk.sda1.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.67311755816819 disk.sda1.in‚0$ disk.nvme0n1.out‚938.51220703125# memory_percent‚97.7086067199707* proc.memory.percent‚16.964316368103027 network.sent‚435753% disk./.usageGB‚1097.7826919555664 disk.sda3.out‚ 10.609375 disk.dm-0.in‚0! disk.dm-0.out‚938.51220703125 disk.dm-1.out‚ 10.609375 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd2.out‚ 17.12890625& proc.memory.availableMB‚ 375.421875 disk.sda.in‚0 disk.nvme0c0n1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2779.43359375 cpu‚25.02645142272597 network.recv‚181910 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.24365234375& disk.nvme0c0n1.out‚938.51220703125 disk.rbd0.in‚0ypºˆ“¦‚(:ˆ ©‚åÔßì„Ž gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»iק‚(:ˆ ©‚åÔîñ¥ disk.dm-1.out‚11.1875$ disk.nvme0n1.out‚938.77783203125 disk.rbd2.in‚0 disk.sda.out‚11.1875 disk.rbd1.out‚ 0.306640625 disk.rbd0.out‚0 disk.rbd2.out‚ 17.2734375 disk.rbd3.out‚0 network.recv‚187848 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd4.in‚0$ memory_percent‚97.71201610565186 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚0* proc.memory.percent‚16.964435577392578 disk.dm-0.in‚0! disk.dm-0.out‚938.77783203125 disk.sda1.out‚0 disk.drbd1777.out‚0( proc.memory.availableMB‚ 374.86328125 network.sent‚486107% disk./.usageGB‚1097.7826347351074 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚938.77783203125 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚11.1875) disk./.usagePercent‚73.67311371805518 disk.sda.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.nvme0n1.in‚0! proc.memory.rssMB‚ 2779.453125 cpu‚25.02765828240309Žažð0¨+ _wandbruntime‚281  _runtime‚2812üÄ©¾ % _timestamp‚1.7884285969227846e+09 _runtime‚ 281.627532694 _step‚28 & lr-AdamW/pg1‚7.479114961071856e-05 & lr-AdamW/pg2‚7.479114961071856e-05 trainer/global_step‚399õØ0¥ÈªÂ _step‚28 & lr-AdamW/pg1‚7.479114961071856e-05 & lr-AdamW/pg2‚7.479114961071856e-05 trainer/global_step‚399 % _timestamp‚1.7884285969227846e+09 _runtime‚ 281.627532694!|K0«+ _wandbruntime‚281  _runtime‚281 xþó°'¬ªO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3272902 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚ 281.628450443 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5613996 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200913920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22886419526144 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 343827471872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 327635905536 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 344004445696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 328291501056 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22886419526144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13036100 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 326430626514 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5112188 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 328291501056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5613996 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22558110022144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24680926436864 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22885745927680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24680950376960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13017558 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176973824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22558110022144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22558128025088 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7422102 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5112666 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21977335612416 trainer/global_step‚399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 327085056722 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7403562 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 327635905536 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944<8ì´'­®O YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 326430626514 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5112188 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 328291501056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5613996 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22558110022144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24680926436864 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22885745927680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24680950376960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13017558 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176973824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22558110022144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22558128025088 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7422102 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5112666 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21977335612416 trainer/global_step‚399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 327085056722 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7403562 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 327635905536 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3272902 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚ 281.628450443 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5613996 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200913920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22886419526144 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 343827471872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 327635905536 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 344004445696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 328291501056 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22886419526144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13036100 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360‚‹¦f0®+ _wandbruntime‚282  _runtime‚282k (Ö¯´M ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22614481063936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1844320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚30 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5628014 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5124932 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3281068 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25087263636480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22943592006144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22614463060992 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24742577174528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚ 282.290571529 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22942918394368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I=DeviceStatsMonitor.on_train_batch_end/actf¦Ô*ive.small_pool.freed‚7422016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22614481063936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13050028 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1843864 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25087454051328 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22943592006144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 344686461952 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 327247061478 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13068596 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 329110942208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13050028 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 327901491894 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7422016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24742601114624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 344852936704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22032256782336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13068596 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5628014 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3281070 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7440582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22360158274230 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22359486804454 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0R•Á»¾&°¸M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7422016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24742601114624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 344852936704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22032256782336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13068596 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5628014 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3281070 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7440582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22360158274230 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22359486804454 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22614481063936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1844320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚30 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5628014 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5124932 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3281068 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25087263636480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22943592006144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22614463060992 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24742577174528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚ 282.290571529 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22942918394368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7422016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22614481063936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13050028 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1843864 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25087454051328 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22943592006144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 344686461952 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 327247061478 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13068596 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 329110942208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13050028 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 327901491894GC·0±+ _wandbruntime‚282  _runtime‚282¸K·AR²Ì /loss/FRt2/assignment_loss‚7.380016395023891 2loss/SRqqt1/assignment_loss‚3.2240783351439015 0loss/FBt1/assignment_loss‚0.6518374636902341 .loss/FRt2/detection_loss‚3.942203027861459 .loss/FBt4/detection_loss‚8.161987806621351 2loss/SRqqt4/assignment_loss‚2.4220445202846155 .loss/FBt2/detection_loss‚4.802844652239591 0loss/FBt2/assignment_loss‚0.7723905630409718 0loss/SRqqt4/detection_loss‚3.316498352032082 .loss/FBt3/detection_loss‚2.199385154501281 _runtime‚ 282.292074731 /loss/FRt4/assignment_loss‚7.272614526748657 1loss/FBt4/assignment_loss‚0.46061190301062244 .loss/FRt4/detection_loss‚4.581232637166977 % _timestamp‚1.7884285975874379e+09 /loss/FRt3/assignment_loss‚7.186176430095326 .loss/FRt1/detection_loss‚3.011738656826739 1loss/SRqqt1/detection_loss‚2.9898883639662355 1loss/SRqqt2/detection_loss‚1.3326985302927457 /loss/FBt1/detection_loss‚11.418603709765843 %loss/total_loss‚93.43291020302122 /loss/FRt1/assignment_loss‚7.394823146316241 1loss/SRqqt2/assignment_loss‚3.270872657497724 2loss/SRqqt3/assignment_loss‚2.8669564784583397 _step‚31 1loss/FBt3/assignment_loss‚0.23203550856454636 /loss/FRt3/detection_loss‚2.5524517057294194 1loss/SRqqt3/detection_loss‚1.9889196781424305 epoch‚0 trainer/global_step‚399Â!‡V³Ð /loss/FRt3/assignment_loss‚7.186176430095326 .loss/FRt1/detection_loss‚3.011738656826739 1loss/SRqqt1/detection_loss‚2.9898883639662355 1loss/SRqqt2/detection_loss‚1.3326985302927457 /loss/FBt1/detection_loss‚11.418603709765843 %loss/total_loss‚93.43291020302122 /loss/FRt1/assignment_loss‚7.394823146316241 1loss/SRqqt2/assignment_loss‚3.270872657497724 2loss/SRqqt3/assignment_loss‚2.8669564784583397 _step‚31 1loss/FBt3/assignment_loss‚0.23203550856454636 /loss/FRt3/detection_loss‚2.5524517057294194 1loss/SRqqt3/detection_loss‚1.9889196781424305 epoch‚0 trainer/global_step‚399 /loss/FRt2/assignment_loss‚7.380016395023891 2loss/SRqqt1/assignment_loss‚3.2240783351439015 0loss/FBt1/assignment_loss‚0.6518374636902341 .loss/FRt2/detection_loss‚3.942203027861459 .loss/FBt4/detection_loss‚8.161987806621351 2loss/SRqqt4/assignment_loss‚2.4220445202846155 .loss/FBt2/detection_loss‚4.802844652239591 0loss/FBt2/assignment_loss‚0.7723905630409718 0loss/SRqqt4/detection_loss‚3.316498352032082 .loss/FBt3/detection_loss‚2.199385154501281 _runtime‚ 282.292074731 /loss/FRt4/assignment_loss‚7.272614526748657 1loss/FBt4/assignment_loss‚0.46061190301062244 .loss/FRt4/detection_loss‚4.581232637166977 % _timestamp‚1.7884285975874379e+09Ù"—Zž´‚(:“ ¸‚åÔδñ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.656' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.731200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹müSµ‚(:“ ¸‚åÔ¤¶˜‘ disk.dm-1.out‚ 11.93359375$ disk.nvme0n1.out‚939.29541015625 disk.rbd1.in‚0 disk.sda3.out‚ 11.93359375) disk./.usagePercent‚73.67311474208532! disk.dm-0.out‚939.29541015625 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚0# memory_percent‚97.7121353149414) proc.memory.percent‚16.96445941925049 disk.drbd1777.in‚0 disk.drbd1777.out‚0& disk.nvme0c0n1.out‚939.29541015625# proc.memory.rssMB‚ 2779.45703125 disk.sda1.out‚0 disk.rbd1.out‚ 0.306640625 disk.sda2.in‚0 disk.sda3.in‚0 cpu‚25.028586255053877% disk./.usageGB‚1097.7826499938965 disk.sda.out‚ 11.93359375 disk.nvme0c0n1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 18.87109375% proc.memory.availableMB‚ 374.84375 network.recv‚191968 disk.dm-0.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 network.sent‚491932 disk.sda1.in‚0¥HX7“¶‚(:ˆ Ç‚åÔîœÌ gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£µi% ·‚(:• Ç‚åÔå°‰ disk.rbd1.in‚0 disk.rbd2.out‚ 19.015625% disk./.usageGB‚1097.7906761169434 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0& disk.nvme0c0n1.out‚939.67431640625$ memory_percent‚97.71344661712646 disk.sda1.out‚0 disk.dm-1.out‚ 12.51953125 disk.nvme0c0n1.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0) disk./.usagePercent‚73.67365338193736 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 0.306640625 disk.rbd0.out‚0( proc.memory.availableMB‚ 374.62890625# proc.memory.rssMB‚ 2779.45703125 network.sent‚542411 disk.rbd3.out‚0! disk.dm-0.out‚939.67431640625 disk.sda.out‚ 12.51953125) proc.memory.percent‚16.96445941925049 cpu‚25.028983086175213 disk.sda3.out‚ 12.51953125 disk.sda.in‚0$ disk.nvme0n1.out‚939.67431640625 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 network.recv‚198214 disk.nvme0n1.in‚0 A˜ž¸‚(:“ Ö‚åÔ‰û gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚101.724' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.344800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"*x¹‚(:‹ Ö‚åÔÏü¬‘ disk.nvme0c0n1.in‚0$ disk.nvme0c0n1.out‚ 948.369140625 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 20.61328125 disk.sda3.out‚ 12.99609375& proc.memory.availableMB‚ 374.765625 disk.sda.in‚0$ disk./.usageGB‚1097.790683746338( proc.memory.percent‚16.9644832611084) disk./.usagePercent‚73.67365389395242 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.out‚ 948.369140625 proc.cpu.threads‚17 network.recv‚202275 disk.drbd1777.out‚0 disk.sda1.out‚0$ memory_percent‚97.71261215209961 cpu‚25.02971373477304 disk.rbd4.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 0.306640625 disk.dm-0.in‚0 disk.sda.out‚ 12.99609375 disk.rbd2.in‚0" proc.memory.rssMB‚ 2779.4609375" disk.nvme0n1.out‚ 948.369140625 network.sent‚548008 disk.drbd1777.in‚0 disk.rbd4.out‚0 disk.dm-1.out‚ 12.99609375 disk.nvme0n1.in‚0 disk.dm-1.in‚0cÏÀ0º+ _wandbruntime‚317  _runtime‚317p'O¸Ä»¾ & lr-AdamW/pg1‚8.416347412333993e-05 & lr-AdamW/pg2‚8.416347412333993e-05 trainer/global_step‚449 % _timestamp‚1.7884286323615918e+09 _runtime‚ 317.066379953 _step‚32Êëȼ _runtime‚ 317.066379953 _step‚32 & lr-AdamW/pg1‚8.416347412333993e-05 & lr-AdamW/pg2‚8.416347412333993e-05 trainer/global_step‚449 % _timestamp‚1.7884286323615918e+09 ùŽ0½+ _wandbruntime‚317  _runtime‚317³ÓÁ"¾ªO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8329614 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8329614 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8348154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28150395387392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚449 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 369277306880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25375761964544 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 368621711360 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8348154 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24722542140416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177845248 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 367265965922 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25744383675904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6314798 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 386932067328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚25089808106338 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 317.067459606 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25745057274368 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 368621711360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytesö’œï.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 367920396130 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6314798 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 369277306880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25375779967488 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25090479575906 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2071980 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ÞÐY1´'¿®O JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚33 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3681302 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27763463320064 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14644410 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6314796 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25375761964544 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25744383675904 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5752830 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25745057274368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5753282 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28150573232640 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27763487260160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 369277306880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25375761964544 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 368621711360 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8348154 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24722542140416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177845248 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 367265965922 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25744383675904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6314798 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 386932067328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚25089808106338 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 317.067459606 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25745057274368 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 368621711360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 367920396130 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6314798 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 369277306880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25375779967488 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25090479575906 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2071980 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8329614 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8329614 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8348154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28150395387392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚449 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8366634 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚25801556142592 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14676880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 370096748032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14695448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25432133006336 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8348068 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5765498 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884286330245008e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25802229754368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 368736831302 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27825114057728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25432115003392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3689470 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14695448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6328814 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8366634 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2076030 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14676880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5765957 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 369441139200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 387788535296 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 369441139200 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 387955010048 _step‚34 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6328812 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24777463310336 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8348068 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25432115003392 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5765957 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 369441139200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 387788535296 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 369441139200 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 387955010048 _step‚34 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6328812 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24777463310336 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8348068 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25432115003392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8366634 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚25801556142592 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14676880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 370096748032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14695448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25432133006336 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8348068 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5765498 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884286330245008e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25802229754368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 368736831302 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27825114057728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25432115003392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3689470 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14695448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6328814 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8366634 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2076030 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944"ÇÌ;³0Ã+ _wandbruntime‚317  _runtime‚317†™ÞLÄÆ /loss/FRt3/assignment_loss‚7.214972161410148 0loss/SRqqt3/assignment_loss‚2.65615625778834 1loss/SRqqt4/assignment_loss‚2.672451379451346 /loss/FRt1/detection_loss‚3.1146861609692373 .loss/FRt3/detection_loss‚2.256997046959505 0loss/SRqqt2/detection_loss‚2.196330188621424 -loss/FBt1/detection_loss‚8.76947518520885 trainer/global_step‚449 1loss/SRqqt3/detection_loss‚2.5545200293534434 .loss/FBt4/detection_loss‚4.670725112961184 _step‚35 /loss/FRt2/detection_loss‚3.4885786585509777 $ _timestamp‚1.788428633026175e+09 1loss/SRqqt1/detection_loss‚2.2465760728809983 epoch‚0 2loss/SRqqt1/assignment_loss‚2.6705535631626844 0loss/FBt2/assignment_loss‚0.6812865998431602 .loss/FRt4/detection_loss‚3.601297826766968 /loss/FBt2/detection_loss‚5.6523840515295065 /loss/FBt3/detection_loss‚1.9537906818772477 2loss/SRqqt2/assignment_loss‚3.4769175781144037 0loss/FBt3/assignment_loss‚0.4671105450227302 %loss/total_loss‚86.85078906213428 _runtime‚ 317.730805531 /loss/FRt1/assignment_loss‚7.376986036698024 /loss/FRt4/assignment_loss‚7.299661121368408 1loss/FBt1/assignment_loss‚0.44091691918826353 0loss/FBt4/assignment_loss‚0.3336783997832343 /loss/SRqqt4/detection_loss‚3.62439578391136 .loss/FRt2/assignment_loss‚7.43034170071284ž4ÚªÅ)£ù´MÊ 2loss/SRqqt1/assignment_loss‚2.6705535631626844 0loss/FBt2/assignment_loss‚0.6812865998431602 .loss/FRt4/detection_loss‚3.601297826766968 /loss/FBt2/detection_loss‚5.6523840515295065 /loss/FBt3/detection_loss‚1.9537906818772477 2loss/SRqqt2/assignment_loss‚3.4769175781144037 0loss/FBt3/assignment_loss‚0.4671105450227302 %loss/total_loss‚86.85078906213428 _runtime‚ 317.730805531 /loss/FRt1/assignment_loss‚7.376986036698024 /loss/FRt4/assignment_loss‚7.299661121368408 1loss/FBt1/assignment_loss‚0.44091691918826353 0loss/FBt4/assignment_loss‚0.3336783997832343 /loss/SRqqt4/detection_loss‚3.62439578391136 .loss/FRt2/assignment_loss‚7.43034170071284 /loss/FRt3/assignment_loss‚7.214972161410148 0loss/SRqqt3/assignment_loss‚2.65615625778834 1loss/SRqqt4/assignment_loss‚2.672451379451346 /loss/FRt1/detection_loss‚3.1146861609692373 .loss/FRt3/detection_loss‚2.256997046959505 0loss/SRqqt2/detection_loss‚2.196330188621424 -loss/FBt1/detection_loss‚8.76947518520885 trainer/global_step‚449 1loss/SRqqt3/detection_loss‚2.5545200293534434 .loss/FBt4/detection_loss‚4.670725112961184 _step‚35 /loss/FRt2/detection_loss‚3.4885786585509777 $ _timestamp‚1.788428633026175e+09 1loss/SRqqt1/detection_loss‚2.2465760728809983 epoch‚0#ŒŽ|ežÆ‚(:“ å‚åÔ—¯ë gpu.0.gpu‚59.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.942' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.188399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§¾¥ÀÇ‚(:† å‚åÔ©Ö¶ disk.rbd2.out‚ 22.328125 disk.drbd1777.out‚0 disk.rbd3.out‚0 network.sent‚554759) disk./.usagePercent‚73.67365542999764 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 13.921875 disk.nvme0c0n1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 network.recv‚207265 disk.dm-0.out‚ 948.705078125 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0% disk./.usageGB‚1097.7907066345215" disk.nvme0n1.out‚ 948.705078125 disk.drbd1777.in‚0' proc.memory.availableMB‚ 374.3984375 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.sda.out‚ 13.921875 disk.nvme0n1.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda2.out‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2779.4609375 cpu‚25.029297555585114$ disk.nvme0c0n1.out‚ 948.705078125 disk.rbd0.in‚0 disk.sda3.out‚ 13.921875( proc.memory.percent‚16.9644832611084 disk.sda.in‚0 disk.rbd1.in‚0$ memory_percent‚97.71485328674316^Ó)B“È‚(:ˆ ô‚åÔç†Õ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚107.959' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü•É'É‚(: ô‚åÔ¹«# disk.nvme0n1.out‚949.4560546875 disk.sda.out‚ 14.41015625 proc.cpu.threads‚17% disk.nvme0c0n1.out‚949.4560546875 network.sent‚604469 disk.dm-1.out‚ 14.41015625 disk.nvme0n1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 22.35546875 disk.sda3.out‚ 14.41015625 disk.rbd4.in‚0 disk.rbd4.out‚0 cpu‚25.030850993940803 disk.rbd3.in‚0 disk.nvme0c0n1.in‚0) disk./.usagePercent‚73.68207449376865 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚97.71184921264648 disk.rbd1.out‚ 0.306640625 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda.in‚0( proc.memory.percent‚16.9644832611084 disk.rbd1.in‚0 disk.sda2.in‚0 disk.dm-0.out‚949.4560546875& proc.memory.availableMB‚ 374.890625 network.recv‚213058% disk./.usageGB‚1097.9161567687988 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚0" proc.memory.rssMB‚ 2779.4609375 disk.dm-1.in‚0`Ç C0Ê+ _wandbruntime‚352  _runtime‚352dæENÄ˾ trainer/global_step‚499 % _timestamp‚1.7884286678231308e+09 _runtime‚ 352.527906363 _step‚36 & lr-AdamW/pg1‚9.353579863596131e-05 & lr-AdamW/pg2‚9.353579863596131e-05Î.ëÈÌ _step‚36 & lr-AdamW/pg1‚9.353579863596131e-05 & lr-AdamW/pg2‚9.353579863596131e-05 trainer/global_step‚499 % _timestamp‚1.7884286678231308e+09 _runtime‚ 352.527906363$ñ†0Í+ _wandbruntime‚352  _runtime‚352;·ú°'ΪO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚499 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 409607517184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16289804 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4089700 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31276304737280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177845248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28603021424128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6392618 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6392179SOìô'Ï®O @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27467748668416 _step‚37 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7015598 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2302916 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 430126688768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2302479 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30846000203264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 408755735538 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7015598 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7015596 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27876521443314 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9274206 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6392618 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6392179 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚499 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 409607517184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16289804 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4089700 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31276304737280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177845248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28603021424128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7029612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16322300 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 430997392384 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6405395 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27932242009046 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9292686 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6404954 % _timestamp‚1.7884286684818811e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 410426945024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30907674881024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16322300 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9274120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28660867502592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28249784948736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7029614 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27931570539270 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 trainer/global_step‚499 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 411082553856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2307086 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 409572170710 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28660867502592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejÊ•6–? ections‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28660193890816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27522669838336 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30907650940928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚38 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31338830359552 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚ 353.187112596 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28249766945792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4097870 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430Àq<}¾&Ò¸M C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16303732 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 411082553856 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16303732 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7029612 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28249784948736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4097868 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27522652798976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16322300 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9274120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28660867502592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28249784948736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7029614 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27931570539270 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 trainer/global_step‚499 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 411082553856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2307086 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 409572170710 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28660867502592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28660193890816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27522669838336 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30907650940928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚38 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31338830359552 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚ 353.187112596 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28249766945792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4097870 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7029612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16322300 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 430997392384 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6405395 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27932242009046 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9292686 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6404954 % _timestamp‚1.7884286684818811e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 410426945024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30907674881024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 473476844032 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7716398 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 451248918528 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 450593323008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31462332770816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10200258 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2534824 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31011083852288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7716396 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10200258 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 451248918528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33928561026560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10181718 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34401860025344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17916656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7032461 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30662563310722 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31011065849344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 449591074946 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31461659172352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33928537086464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10181718 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7716398 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 473322938880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30212972235776 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31461659172352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2534361 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4498102 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17898114 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7032926 trainer/global_step‚549 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31462332770816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34402037870592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177845248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚41 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30212955196416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 451248918528 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 450593323008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31462332770816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10200258 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2534824 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31011083852288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7716396 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10200258 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 451248918528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33928561026560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10181718 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34401860025344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17916656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7032461 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30662563310722 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [HDeviceStatsMonitor.on_train_batch_start/allocat½W%,ÿed_bytes.large_pool.freed‚31011065849344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 449591074946 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31461659172352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33928537086464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10181718 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7716398 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 473322938880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30212972235776 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31461659172352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2534361 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4498102 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17898114 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7032926 trainer/global_step‚549 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31462332770816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34402037870592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177845248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚41 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30212955196416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 473476844032 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7716398 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33990187824128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31067436891136 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31067418888192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30267876366336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 474196957696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4506270 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31519505250816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 450407510118 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 449753079702 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10200172 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17949152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33990211764224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17949152 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31067418888192 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7045659 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7045194 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31519505250816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177831936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31067436891136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10200172 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚42 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 474350849536 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2539389 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34464384781824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 451412750848 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7730412 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10218738 _runtime‚ 388.472240631 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 452068359680 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34464562613760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153891840 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 451412750848 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31518831639040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30717612406678 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚549 % _timestamp‚1.7884287037669673e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7730412 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17930584 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17930584ðÆ¡mÂ&â¼M OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7045194 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31519505250816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177831936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31067436891136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10200172 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚42 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 474350849536 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2539389 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34464384781824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 451412750848 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7730412 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10218738 _runtime‚ 388.472240631 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 452068359680 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34464562613760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153891840 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 451412750848 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31518831639040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30717612406678 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚549 % _timestamp‚1.7884287037669673e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7730412 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17930584 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17930584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4506268 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2538926 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33990187824128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31067436891136 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31067418888192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30267876366336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 474196957696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4506270 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31519505250816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 450407510118 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 449753079702 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10200172 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17949152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33990211764224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17949152 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31067418888192 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7045659 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0*C. 0ã+ _wandbruntime‚388  _runtime‚388 •úIOäÉ 2loss/SRqqt3/assignment_loss‚2.8198513948000397 0loss/FBt2/assignment_loss‚0.6623022609523365 /loss/FRt3/detection_loss‚2.8753158200380144 % _timestamp‚1.7884287037686093e+09 /loss/FBt4/assignment_loss‚0.372018449210979 /loss/FRt2/detection_loss‚3.5734207729498544 /loss/FRt1/detection_loss‚2.5943309776484966 /loss/FRt1/assignment_loss‚6.093121941884359 0loss/SRqqt2/detection_loss‚3.302899225517344 .loss/FBt1/detection_loss‚7.709715178608894 %loss/total_loss‚96.79011789473321 1loss/SRqqt1/detection_loss‚2.6588753304547734 epoch‚0 /loss/FRt4/assignment_loss‚7.332760214805603 1loss/SRqqt1/assignment_loss‚2.660513745413886 0loss/FBt3/assignment_loss‚0.3812282992176933 .loss/FRt4/detection_loss‚4.943514585494995 _runtime‚ 388.473252968 _step‚43 /loss/FRt2/assignment_loss‚7.375850094689263 /loss/FBt1/assignment_loss‚0.305156437685946 1loss/SRqqt4/detection_loss‚3.5323354601860046 trainer/global_step‚549 2loss/SRqqt2/assignment_loss‚3.0551482583537246 2loss/SRqqt4/assignment_loss‚2.8631044727690678 .loss/FBt4/detection_loss‚5.175381402450579 .loss/FRt3/assignment_loss‚7.20910120983513 1loss/SRqqt3/detection_loss‚1.4241212197579443 .loss/FBt2/detection_loss‚16.47536794628416 /loss/FBt3/detection_loss‚1.3946831957241137%EAÛSåÍ /loss/FRt1/detection_loss‚2.5943309776484966 /loss/FRt1/assignment_loss‚6.093121941884359 0loss/SRqqt2/detection_loss‚3.302899225517344 .loss/FBt1/detection_loss‚7.709715178608894 %loss/total_loss‚96.79011789473321 1loss/SRqqt1/detection_loss‚2.6588753304547734 epoch‚0 /loss/FRt4/assignment_loss‚7.332760214805603 1loss/SRqqt1/assignment_loss‚2.660513745413886 0loss/FBt3/assignment_loss‚0.3812282992176933 .loss/FRt4/detection_loss‚4.943514585494995 _runtime‚ 388.473252968 _step‚43 /loss/FRt2/assignment_loss‚7.375850094689263 /loss/FBt1/assignment_loss‚0.305156437685946 1loss/SRqqt4/detection_loss‚3.5323354601860046 trainer/global_step‚549 2loss/SRqqt2/assignment_loss‚3.0551482583537246 2loss/SRqqt4/assignment_loss‚2.8631044727690678 .loss/FBt4/detection_loss‚5.175381402450579 .loss/FRt3/assignment_loss‚7.20910120983513 1loss/SRqqt3/detection_loss‚1.4241212197579443 .loss/FBt2/detection_loss‚16.47536794628416 /loss/FBt3/detection_loss‚1.3946831957241137 2loss/SRqqt3/assignment_loss‚2.8198513948000397 0loss/FBt2/assignment_loss‚0.6623022609523365 /loss/FRt3/detection_loss‚2.8753158200380144 % _timestamp‚1.7884287037686093e+09 /loss/FBt4/assignment_loss‚0.372018449210979 /loss/FRt2/detection_loss‚3.5734207729498544+ŽàïZæ‚(:’ ¡ƒåÔåñ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.045999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09¸iç‚(:‘ ¡ƒåÔ¾”‘% disk.nvme0n1.out‚1148.01416015625 disk.sda2.in‚0 disk.rbd2.in‚0" disk.dm-0.out‚1148.01416015625 disk.sda.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625 disk.rbd2.out‚25.75 disk.drbd1777.out‚0 proc.memory.rssMB‚ 2779.46875 disk.rbd4.out‚0 disk.nvme0n1.in‚0 disk.sda3.out‚ 16.48828125 disk.sda1.in‚0) proc.memory.percent‚16.96453094482422 disk.nvme0c0n1.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 16.48828125 disk.drbd1777.in‚0$ memory_percent‚97.71525859832764 network.recv‚228377) disk./.usagePercent‚73.70728150760502' disk.nvme0c0n1.out‚1148.01416015625 disk.sda2.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 374.33203125 disk.dm-0.in‚0 network.sent‚667102% disk./.usageGB‚1098.2917594909668 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda.out‚ 16.48828125 cpu‚25.032192504142706€Ý/¼’è‚(:‡ °ƒåÔñ䈎 gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.303' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.4606 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µÅ²Jé‚(:‘ °ƒåÔÖÎé disk.dm-0.in‚0 disk.dm-1.out‚ 17.2578125% disk.nvme0n1.out‚1247.67333984375' disk.nvme0c0n1.out‚1247.67333984375 disk.sda2.in‚0 proc.memory.rssMB‚ 2779.46875 cpu‚25.03285216011443$ disk./.usageGB‚1098.290325164795 disk.rbd2.out‚ 25.77734375 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.nvme0n1.in‚0 disk.sda.out‚ 17.2578125 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd4.out‚0 disk.rbd1.out‚ 0.306640625 proc.cpu.threads‚17 network.recv‚234179) disk./.usagePercent‚73.70718524877215 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd0.in‚0 network.sent‚717042 disk.sda3.in‚0 disk.rbd3.out‚0" disk.dm-0.out‚1247.67333984375 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 374.73046875 disk.sda3.out‚ 17.2578125 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda2.out‚0# memory_percent‚97.7128267288208) proc.memory.percent‚16.96453094482422‰U$ªžê‚(:“ ¿ƒåÔ‹ú gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.811' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.362199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pP ë‚(:Œ ¿ƒåÔÀ¢§ disk.sda3.out‚18.15625 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda2.in‚0 disk.rbd2.out‚ 27.4921875 disk.sda.in‚0 disk.sda.out‚18.15625 disk.sda1.out‚0 disk.rbd1.out‚ 0.306640625 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.dm-1.out‚18.15625% disk.nvme0n1.out‚1350.10205078125 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd4.in‚0) proc.memory.percent‚16.96453094482422 cpu‚25.033489143076284) disk./.usagePercent‚73‘|¸Èø.70739184685227 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0' proc.memory.availableMB‚ 374.4765625 network.sent‚723591 network.recv‚239082 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚0" disk.dm-0.out‚1350.10205078125 disk.dm-0.in‚0$ memory_percent‚97.71437644958496 proc.memory.rssMB‚ 2779.46875% disk./.usageGB‚1098.2934036254883 disk.nvme0n1.in‚0' disk.nvme0c0n1.out‚1350.10205078125 disk.sda2.out‚0 disk.sda3.in‚0¸mÏ0ì+ _wandbruntime‚423  _runtime‚423é¬þÆíÀ _runtime‚ 423.452345014 _step‚44 ' lr-AdamW/pg1‚0.00011228044766120406 ' lr-AdamW/pg2‚0.00011228044766120406 trainer/global_step‚599 % _timestamp‚1.7884287387476037e+09S¿.”ÊîÄ trainer/global_step‚599 % _timestamp‚1.7884287387476037e+09 _runtime‚ 423.452345014 _step‚44 ' lr-AdamW/pg1‚0.00011228044766120406 ' lr-AdamW/pg2‚0.00011228044766120406,½¥æt0ï+ _wandbruntime‚423  _runtime‚423ñQ<†´'ð®O PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11107770 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 169456640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33828735794688 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33828717791744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19524966 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37011097909760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8417196 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 491579128832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33447933708562 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32958161724416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 trainer/global_step‚599 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33828717791744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4906502 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34320296920576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33828735794688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚26054ŽX«¸'ñ²O VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 491579128832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33447933708562 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32958161724416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 trainer/global_step‚599 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33828717791744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4906502 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34320296920576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33828735794688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11107770 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 169456640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33828735794688 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33828717791744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19524966 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37011097909760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8417196 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096-Ze_k0ò+ _wandbruntime‚424  _runtime‚424NyÎe¾&ó¸M NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7686247 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19576004 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33013082894336 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19576004 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 492398556672 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 492398556672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 490588419110 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚ 424.127616317 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 493054165504 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4914670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33503654274086 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19557436 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11144790 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11126224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34377469387264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11126224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7685773 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11144790 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151794688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11126224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_poo̼ÃJ!l.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7685773 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11144790 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151794688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19576004 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33013082894336 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19576004 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 492398556672 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 492398556672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 490588419110 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚ 424.127616317 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 493054165504 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4914670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33503654274086 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19557436 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11144790 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11126224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34377469387264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21170360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37179608267264 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21151818 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36646387737088 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8312520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚49 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 560095342080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37179608267264 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 533220530176 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5314900 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37178934668800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36234647045538 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36646387737088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚649 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8312061 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12052362 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37178934668800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 533220530176Ý3Ϭf²O D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21151818 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36646387737088 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8312520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚49 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 560095342080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37179608267264 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 533220530176 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5314900 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37178934668800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36234647045538 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36646387737088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚649 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8312061 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12052362 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37178934668800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 533220530176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21170360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37179608267264 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884287745752563e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21184288 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3002237 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 560811971584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37236780747264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36289696141494 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12070842 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21202856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40716073562112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 532078188934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9132012 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 531423758518 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚ 459.280582049 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40716251394048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36290367611270 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36702722772992 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9132012 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12052276 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5323068 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 533384362496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8324857 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3001789 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5323070 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0öâ`Â&„¼M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 534039971328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21184288 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9132014 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35758272382976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚649 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884287745752563e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21184288 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3002237 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 560811971584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37236780747264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36289696141494 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12070842 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21202856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40716073562112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 532078188934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9132012 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 531423758518 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚ 459.280582049 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40716251394048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36290367611270 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36702722772992 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9132012 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12052276 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5323068 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 533384362496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8324857 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3001789 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5323070 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 101942558722%+ 0…+ _wandbruntime‚459  _runtime‚459F£§‹O†É % _timestamp‚1.7884287745771947e+09 _runtime‚ 459.282182524 /loss/FRt3/assignment_loss‚7.200050183704922 /loss/FRt4/assignment_loss‚5.166291543415615 /loss/FBt4/assignment_loss‚0.583771001499877 .loss/FRt2/detection_loss‚4.007032279227231 /loss/FBt1/detection_loss‚1.8967532576061785 0loss/FBt3/assignment_loss‚0.7047331625166408 .loss/FRt3/detection_loss‚4.951141466120524 1loss/SRqqt3/assignment_loss‚2.764269971847534 0loss/SRqqt2/detection_loss‚4.541411629680431 0loss/FBt1/assignment_loss‚0.2951296366675251 1loss/SRqqt1/detection_loss‚1.8329343731338914 /loss/FBt3/detection_loss‚5.4382081175793955 .loss/FBt4/detection_loss‚7.992332840959231 %loss/total_loss‚86.17712096967723 epoch‚0 _step‚51 /loss/FRt2/assignment_loss‚6.887457280545621 /loss/FRt4/detection_loss‚2.9107641790594374 2loss/SRqqt1/assignment_loss‚2.2855658353985966 .loss/FRt1/detection_loss‚2.323567588692126 0loss/SRqqt3/detection_loss‚3.914659582823515 0loss/SRqqt4/detection_loss‚6.195071872472763 trainer/global_step‚649 2loss/SRqqt2/assignment_loss‚2.9633927489771987 0loss/FBt2/assignment_loss‚0.6710197307426521 /loss/FRt1/assignment_loss‚4.450345364169798 2loss/SRqqt4/assignment_loss‚2.6232652711868285 /loss/FBt2/detection_loss‚3.5779520516496954&#‹ÕS‡Í 2loss/SRqqt2/assignment_loss‚2.9633927489771987 0loss/FBt2/assignment_loss‚0.6710197307426521 /loss/FRt1/assignment_loss‚4.450345364169798 2loss/SRqqt4/assignment_loss‚2.6232652711868285 /loss/FBt2/detection_loss‚3.5779520516496954 % _timestamp‚1.7884287745771947e+09 _runtime‚ 459.282182524 /loss/FRt3/assignment_loss‚7.200050183704922 /loss/FRt4/assignment_loss‚5.166291543415615 /loss/FBt4/assignment_loss‚0.583771001499877 .loss/FRt2/detection_loss‚4.007032279227231 /loss/FBt1/detection_loss‚1.8967532576061785 0loss/FBt3/assignment_loss‚0.7047331625166408 .loss/FRt3/detection_loss‚4.951141466120524 1loss/SRqqt3/assignment_loss‚2.764269971847534 0loss/SRqqt2/detection_loss‚4.541411629680431 0loss/FBt1/assignment_loss‚0.2951296366675251 1loss/SRqqt1/detection_loss‚1.8329343731338914 /loss/FBt3/detection_loss‚5.4382081175793955 .loss/FBt4/detection_loss‚7.992332840959231 %loss/total_loss‚86.17712096967723 epoch‚0 _step‚51 /loss/FRt2/assignment_loss‚6.887457280545621 /loss/FRt4/detection_loss‚2.9107641790594374 2loss/SRqqt1/assignment_loss‚2.2855658353985966 .loss/FRt1/detection_loss‚2.323567588692126 0loss/SRqqt3/detection_loss‚3.914659582823515 0loss/SRqqt4/detection_loss‚6.195071872472763 trainer/global_step‚6493]jUwžˆ‚(:“ ìƒåÔ¾ˆŽ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚131.617' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.323399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0._鉂(:Œ ìƒåÔ ÂÐ) disk./.usagePercent‚73.74098617952542 disk.dm-0.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚0 disk.sda2.out‚0 cpu‚25.035477947135263 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚1549.50634765625 disk.rbd0.out‚0$ memory_percent‚97.71554470062256 network.recv‚253691% disk./.usageGB‚1098.7939834594727" disk.dm-0.out‚1549.50634765625 disk.dm-1.out‚19.9375 disk.rbd2.out‚ 30.765625( proc.memory.availableMB‚ 374.28515625 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda.out‚19.9375 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd3.in‚0 disk.rbd4.in‚0' disk.nvme0c0n1.out‚1549.50634765625 disk.rbd2.in‚0# proc.memory.rssMB‚ 2779.48046875) proc.memory.percent‚16.96460247039795 network.sent‚785345 disk.sda3.out‚19.9375 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 0.306640625 disk.sda2.in‚0·õ>“Š‚(:ˆ ûƒåÔâ«÷ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.039' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-½KÓ"‹‚(:— ûƒåÔäò¼ disk.sda3.out‚ 20.72265625 disk.rbd4.in‚0 disk.rbd2.in‚0% disk./.usageGB‚1098.7939071655273 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 cpu‚25.035429611619282 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.rbd1.out‚ 0.306640625) proc.memory.percent‚16.96460247039795) disk./.usagePercent‚73.74098105937475 disk.drbd1777.in‚0 disk.rbd4.out‚0% disk.nvme0n1.out‚1648.65283203125' disk.nvme0c0n1.out‚1648.65283203125 disk.sda2.out‚0# proc.memory.rssMB‚ 2779.48046875 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd2.out‚30.90625$ memory_percent‚97.71525859832764 disk.dm-1.out‚ 20.72265625 disk.nvme0c0n1.in‚0 disk.drbd1777.out‚0" disk.dm-0.out‚1648.65283203125 disk.rbd1.in‚0 network.sent‚835705 network.recv‚259839 disk.sda.out‚ 20.72265625 disk.sda2.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 374.33203125Öa¨0Œ+ _wandbruntime‚494  _runtime‚494J.ðľ trainer/global_step‚699 % _timestamp‚1.7884288093042138e+09 _runtime‚ 494.008949196 _step‚52 & lr-AdamW/pg1‚0.0001310250966864468 & lr-AdamW/pg2‚0.0001310250966864468 ½e.ÈŽÂ & lr-AdamW/pg1‚0.0001310250966864468 & lr-AdamW/pg2‚0.0001310250966864468 trainer/global_step‚699 % _timestamp‚1.7884288093042138e+09 _runtime‚ 494.008949196 _step‚524 Ëê0+ _wandbruntime‚494  _runtime‚494Ò¼½f®O LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40038246015488 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 603449187328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12959874 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9818798 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12978414 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39464039679488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39464039679488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40037572417024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5723302 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 156002304 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 603293185024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40037572417024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 573550740480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 573550740480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39464021676544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3230797 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚699 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22778670 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5723300 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38448591819776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bÀ]ÑNytes.all.peak‚ 9954819072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39464021676544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3230334 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39020017443378 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9818798 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9818796 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39020688912946 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 574206336000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12959874 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43176171676160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚53 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12978414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414Çú‚Ǹ'‘²O O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40037572417024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 573550740480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 573550740480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39464021676544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3230797 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚699 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22778670 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5723300 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38448591819776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39464021676544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3230334 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39020017443378 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9818798 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9818796 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39020688912946 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 574206336000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12959874 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43176171676160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚53 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12978414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40038246015488 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 603449187328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12959874 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9818798 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12978414 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39464039679488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39464039679488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40037572417024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5723302 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 156002304 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6032931850245/ Ø0’+ _wandbruntime‚494  _runtime‚494Ê ¾&“¸M QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3234882 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22811140 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8966827 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39520392718336 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 575025777152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43841949377024 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 604150903296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚699 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 574370168320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38503495950336 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39076409478678 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 572913528342 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22829708 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39520392718336 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43237822413824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12996894 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39520374715392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12996894 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12978328 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39075738008902 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9832814 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604317378048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39520374715392 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9832814 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 575025777152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚477 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22811140 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40094744883712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9832812 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40095418495488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22829708 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 572259097926 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8966350 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚ 494.663730127 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40095418495488 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5731470 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚475 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38503478910976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43237798473728 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9832812 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884288099585767e+09 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 574370168320 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5731468 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704Ü9UM锼M I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9832812 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884288099585767e+09 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 574370168320 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5731468 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3234882 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22811140 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8966827 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39520392718336 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 575025777152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43841949377024 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 604150903296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚699 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 574370168320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38503495950336 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39076409478678 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 572913528342 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22829708 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39520392718336 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43237822413824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12996894 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39520374715392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12996894 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12978328 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39075738008902 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9832814 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604317378048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39520374715392 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9832814 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 575025777152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚477 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22811140 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40094744883712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9832812 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40095418495488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22829708 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 572259097926 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8966350 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚ 494.663730127 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40095418495488 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5731470 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚475 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38503478910976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚432377984737286U#0•+ _wandbruntime‚494  _runtime‚494b³]ŸO–É .loss/FBt3/detection_loss‚4.189797017915241 /loss/FRt2/assignment_loss‚6.654348757959181 /loss/FRt3/assignment_loss‚6.269526861151871 .loss/FBt1/detection_loss‚2.509598776301805 _runtime‚ 494.66467601 .loss/FRt1/assignment_loss‚4.17187683398907 1loss/FBt3/assignment_loss‚0.18437115053762737 .loss/FBt2/detection_loss‚5.707090191543102 epoch‚0 _step‚55 /loss/FRt4/assignment_loss‚6.263622113636562 2loss/SRqqt4/assignment_loss‚3.0445932410657406 %loss/total_loss‚85.04953153446755 % _timestamp‚1.7884288099600327e+09 0loss/FBt2/assignment_loss‚0.6006970265880227 .loss/FRt2/detection_loss‚4.582932649120208 /loss/FRt4/detection_loss‚3.9871741448129927 trainer/global_step‚699 1loss/SRqqt1/detection_loss‚1.3353168753612983 1loss/SRqqt2/detection_loss‚3.5508625768125057 0loss/SRqqt3/detection_loss‚4.395860091290054 .loss/FBt4/detection_loss‚9.959433481097221 .loss/FRt3/detection_loss‚2.723584698475137 1loss/SRqqt1/assignment_loss‚2.410771282095658 2loss/SRqqt2/assignment_loss‚2.8889504756246294 2loss/SRqqt3/assignment_loss‚2.3576631002566395 0loss/FBt1/assignment_loss‚0.1665873387173633 1loss/FBt4/assignment_loss‚0.38560608153823395 /loss/FRt1/detection_loss‚2.0308523474213405 0loss/SRqqt4/detection_loss‚4.678414421156049N {®S—Í .loss/FRt3/detection_loss‚2.723584698475137 1loss/SRqqt1/assignment_loss‚2.410771282095658 2loss/SRqqt2/assignment_loss‚2.8889504756246294 2loss/SRqqt3/assignment_loss‚2.3576631002566395 0loss/FBt1/assignment_loss‚0.1665873387173633 1loss/FBt4/assignment_loss‚0.38560608153823395 /loss/FRt1/detection_loss‚2.0308523474213405 0loss/SRqqt4/detection_loss‚4.678414421156049 .loss/FBt3/detection_loss‚4.189797017915241 /loss/FRt2/assignment_loss‚6.654348757959181 /loss/FRt3/assignment_loss‚6.269526861151871 .loss/FBt1/detection_loss‚2.509598776301805 _runtime‚ 494.66467601 .loss/FRt1/assignment_loss‚4.17187683398907 1loss/FBt3/assignment_loss‚0.18437115053762737 .loss/FBt2/detection_loss‚5.707090191543102 epoch‚0 _step‚55 /loss/FRt4/assignment_loss‚6.263622113636562 2loss/SRqqt4/assignment_loss‚3.0445932410657406 %loss/total_loss‚85.04953153446755 % _timestamp‚1.7884288099600327e+09 0loss/FBt2/assignment_loss‚0.6006970265880227 .loss/FRt2/detection_loss‚4.582932649120208 /loss/FRt4/detection_loss‚3.9871741448129927 trainer/global_step‚699 1loss/SRqqt1/detection_loss‚1.3353168753612983 1loss/SRqqt2/detection_loss‚3.5508625768125057 0loss/SRqqt3/detection_loss‚4.395860091290054 .loss/FBt4/detection_loss‚9.9594334810972217Gòw“˜‚(:ˆ Š„åÔ³€ò gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.056' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø2]Ë™‚(:Š Š„åÔ¼òŠ‘ disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 cpu‚25.03578964648345 disk.dm-1.out‚21.4375) disk./.usagePercent‚73.74099437176652 disk.dm-0.in‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚1748.1650390625 disk.sda3.out‚21.4375 disk.sda.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 374.27734375% disk./.usageGB‚1098.7941055297852 disk.sda1.out‚0 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚1748.1650390625 disk.rbd2.in‚0 disk.rbd1.out‚ 0.306640625 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.rbd4.out‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd2.out‚ 32.48046875# proc.memory.rssMB‚ 2779.48046875) proc.memory.percent‚16.96460247039795 disk.rbd4.in‚0! disk.dm-0.out‚1748.1650390625 disk.sda.out‚21.4375 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0$ memory_percent‚97.71559238433838 proc.cpu.threads‚17 network.sent‚840378 network.recv‚263046 disk.rbd0.out‚0K!U~žš‚(:“ ™„åÔ““‘Ž gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.692' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.138400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸Š±Z›‚(:“ ™„åԱ麑 disk.rbd0.out‚0) disk./.usagePercent‚73.74098925161584 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.rbd1.out‚ 0.306640625! proc.memory.rssMB‚ 2779.484375 proc.cpu.threads‚17 disk.dm-1.out‚ 22.2734375" disk.dm-0.out‚1847.41552734375 disk.nvme0n1.in‚0 disk.sda.out‚ 22.2734375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd2.out‚ 32.65234375 network.sent‚890853 network.recv‚269337 disk.rbd4.out‚0 disk.sda.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0$ memory_percent‚97.71738052368164% disk./.usageGB‚1098.7940292358398% disk.nvme0n1.out‚1847.41552734375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 22.2734375 disk.drbd1777.out‚0 disk.rbd3.out‚0 cpu‚25.036148835967843 disk.sda1.in‚0 disk.dm-0.in‚0' disk.nvme0c0n1.out‚1847.41552734375 disk.sda3.in‚0 disk.rbd3.in‚0& proc.memory.availableMB‚ 373.984375) proc.memory.percent‚16.96462631225586 disk.sda2.out‚0è°çŒ’œ‚(:‡ ¨„åÔ®å gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚88.219' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6438 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Êq‚(:’ ¨„åÔÝÎœ network.recv‚273436 disk.rbd4.out‚0 disk.sda.out‚ 22.73046875 disk.dm-1.out‚ 22.73046875$ disk.nvme0n1.out‚1848.0732421875 disk.rbd1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.74099309172884 disk.rbd2.in‚0 disk.rbd2.out‚ 34.2265625 disk.rbd4.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 0.306640625 disk.rbd3.in‚0! disk.dm-0.out‚1848.0732421875 disk.dm-1.in‚0& disk.nvme0c0n1.out‚1848.0732421875 disk.nvme0c0n1.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2779.484375 disk.rbd0.out‚0 disk.sda3.out‚ 22.73046875 disk.drbd1777.out‚0 disk.sda.in‚0 cpu‚25.03618994996114% disk./.usageGB‚1098.7940864562988 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.sda2.in‚0$ memory_percent‚97.71475791931152' proc.memory.availableMB‚ 374.4140625 network.sent‚896501 disk.drbd1777.in‚0) proc.memory.percent‚16.96462631225586È.²]0ž+ _wandbruntime‚529  _runtime‚529­hxZÆŸÀ % _timestamp‚1.7884288444068105e+09 _runtime‚ 529.111598152 _step‚56 ' lr-AdamW/pg1‚0.00014039742119906817 ' lr-AdamW/pg2‚0.00014039742119906817 trainer/global_step‚749Ý ¦%Ê Ä ' lr-AdamW/pg1‚0.00014039742119906817 ' lr-AdamW/pg2‚0.00014039742119906817 trainer/global_step‚749 % _timestamp‚1.7884288444068105e+09 _runtime‚ 529.111598152 _step‚568'´è0¡+ _wandbruntime‚529  _runtime‚529 ­æ·'¢±O IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 612278002370 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41806059310786 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10519598 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10519596 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9594514 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 646587082240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42896883763712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46258684619264 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 614536546304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3462812 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 156002304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9594046 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42281691621888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42281673618944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10519596 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6131702 _step‚57 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615192141824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13904466 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚ 529.11261127 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42281673618944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 614536546304 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41193781308416 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42896210165248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42896883763712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 615192141824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24424064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3462346 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 612932432578 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6131700 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13885926 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 646431079936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179942400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844±^Ƭ»'£µO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24424064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3462346 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 612932432578 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6131700 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13885926 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 646431079936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179942400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 612278002370 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41806059310786 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10519598 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10519596 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9594514 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 646587082240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42896883763712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46258684619264 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 614536546304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3462812 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 156002304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9594046 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42281691621888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42281673618944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10519596 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6131702 _step‚57 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615192141824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13904466 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚ 529.11261127 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42281673618944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 614536546304 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41193781308416 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42896210165248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42896883763712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6151921418249—”0¤+ _wandbruntime‚529  _runtime‚529¼P‡¸2¥¼M NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10533614 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24456560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 % _timestamp‚1.7884288450725038e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10533612 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42338026657792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24437992 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42953382631936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41862451346086 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41248685438976 trai s­!ner/global_step‚749 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 184123392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24456560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9607350 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3467480 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6139868 _step‚58 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 616011582976 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9606872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13922946 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42954056243712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 160183296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 616011582976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42338026657792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42954056243712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24437992 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6139870 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42338044660736 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10533612 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 613094437334 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872Àì;‹Æ&¦ÀM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6139870 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42338044660736 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10533612 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 613094437334 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10533614 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24456560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 % _timestamp‚1.7884288450725038e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10533612 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42338026657792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24437992 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42953382631936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41862451346086 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41248685438976 trainer/global_step‚749 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 184123392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24456560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9607350 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3467480 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6139868 _step‚58 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 616011582976 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9606872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13922946 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42954056243712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 160183296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 616011582976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42338026657792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42954056243712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24437992 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45754847913472 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 653113341778 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26032374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14830518 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 564.26341232 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14811978 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6540102 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49341245442560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45755521511936 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43938987836416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45755521511936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 653767771986 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 656177947648 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45099343564288 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 655522352128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45099325561344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50030975280640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 182039552 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43939004875776 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3694845 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 689729838080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45099325561344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11220396 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6540100 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14811978 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884288795584197e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26050916 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3694379 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45099343564288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 L¯d5žBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚61 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10234479 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45754847913472 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 653113341778 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26032374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14830518 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 564.26341232 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14811978 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6540102 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49341245442560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45755521511936 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43938987836416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45755521511936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 653767771986 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 656177947648 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45099343564288 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 655522352128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45099325561344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50030975280640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 182039552 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43939004875776 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3694845 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 689729838080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45099325561344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11220396 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6540100 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14811978 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884288795584197e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26050916 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3694379 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45099343564288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚61 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10234479 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45812693991936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49402896180224 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44647821743718 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45812020380160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3699443 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43993891966976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26064844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11234414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10247250 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6548270 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 690603436544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45155696603136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26064844 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14830432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45812693991936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179929088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6548268 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45155696603136 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 656341779968 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50093319687680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26083412 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 653929776742 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45155678600192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14848998 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚ 564.927179867 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10247713 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45155678600192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50093499616768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49402872240128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155988992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OºÈ&¶ÂM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11234412 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45812693991936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49402896180224 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44647821743718 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45812020380160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3699443 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43993891966976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26064844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11234414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10247250 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6548270 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 690603436544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45155696603136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26064844 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14830432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45812693991936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179929088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6548268 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45155696603136 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 656341779968 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50093319687680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26083412 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 653929776742 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45155678600192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14848998 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚ 564.927179867 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10247713 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45155678600192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50093499616768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49402872240128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155988992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0>òâ±è0·+ _wandbruntime‚564  _runtime‚564sÅ«ûˆ¸Í .loss/FRt1/detection_loss‚2.492074174228413 /loss/FRt2/detection_loss‚3.4739161620014594 .loss/FRt4/detection_loss‚3.182500538073088 /loss/FBt2/detection_loss‚10.359211213058895 _runtime‚ 564.928289067 .loss/FBt3/detection_loss‚7.829137908270961 /loss/FRt1/assignment_loss‚4.211257048721971 /loss/FRt4/assignment_loss‚5.867575193706312 2loss/SRqqt3/as>‹6ïËsignment_loss‚2.2802502814420436 /loss/FRt3/detection_loss‚3.3697357933188594 1loss/SRqqt2/detection_loss‚3.4065257719694038 /loss/FBt1/detection_loss‚1.2576007131469478 2loss/SRqqt1/assignment_loss‚2.7114034805032943 1loss/SRqqt2/assignment_loss‚3.015664186233129 0loss/FBt3/assignment_loss‚0.7409181042263905 0loss/SRqqt3/detection_loss‚2.990729101868563 %loss/total_loss‚83.99226598018203 trainer/global_step‚799 2loss/SRqqt4/assignment_loss‚2.1630836967378855 0loss/FBt1/assignment_loss‚0.3229940239301063 epoch‚0 % _timestamp‚1.7884288802236495e+09 _step‚63 /loss/FRt3/assignment_loss‚5.464846577755241 0loss/FBt4/assignment_loss‚0.3962462506339332 .loss/FBt4/detection_loss‚6.929401992095841 /loss/FRt2/assignment_loss‚6.070353068803486 1loss/SRqqt1/detection_loss‚1.0908601335456802 1loss/FBt2/assignment_loss‚0.18800526986726457 1loss/SRqqt4/detection_loss‚4.1779752960428596ïî¥W¹Ñ .loss/FRt4/detection_loss‚3.182500538073088 /loss/FBt2/detection_loss‚10.359211213058895 _runtime‚ 564.928289067 .loss/FBt3/detection_loss‚7.829137908270961 /loss/FRt1/assignment_loss‚4.211257048721971 /loss/FRt4/assignment_loss‚5.867575193706312 2loss/SRqqt3/assignment_loss‚2.2802502814420436 /loss/FRt3/detection_loss‚3.3697357933188594 1loss/SRqqt2/detection_loss‚3.4065257719694038 /loss/FBt1/detection_loss‚1.2576007131469478 2loss/SRqqt1/assignment_loss‚2.7114034805032943 1loss/SRqqt2/assignment_loss‚3.015664186233129 0loss/FBt3/assignment_loss‚0.7409181042263905 0loss/SRqqt3/detection_loss‚2.990729101868563 %loss/total_loss‚83.99226598018203 trainer/global_step‚799 2loss/SRqqt4/assignment_loss‚2.1630836967378855 0loss/FBt1/assignment_loss‚0.3229940239301063 epoch‚0 % _timestamp‚1.7884288802236495e+09 _step‚63 /loss/FRt3/assignment_loss‚5.464846577755241 0loss/FBt4/assignment_loss‚0.3962462506339332 .loss/FBt4/detection_loss‚6.929401992095841 /loss/FRt2/assignment_loss‚6.070353068803486 1loss/SRqqt1/detection_loss‚1.0908601335456802 1loss/FBt2/assignment_loss‚0.18800526986726457 1loss/SRqqt4/detection_loss‚4.1779752960428596 .loss/FRt1/detection_loss‚2.492074174228413 /loss/FRt2/detection_loss‚3.4739161620014594?[0“º‚(:ˆ Õ„åÔê ê gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.3156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º >»‚(:‰ Õ„åÔݼÞ network.sent‚959250) disk./.usagePercent‚73.74098541150282 disk.rbd1.out‚ 0.306640625 disk.sda2.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚37.625 disk.rbd3.out‚0 disk.dm-1.out‚ 24.765625 disk.rbd1.in‚0& disk.nvme0c0n1.out‚2145.5810546875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd4.out‚0) proc.memory.percent‚16.96462631225586 disk.rbd0.out‚0 disk.sda3.out‚ 24.765625 disk.rbd4.in‚0! proc.memory.rssMB‚ 2779.484375 cpu‚25.037188242791522 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚0 disk.sda1.out‚0# memory_percent‚97.7184534072876( proc.memory.availableMB‚ 373.80859375 network.recv‚288885% disk./.usageGB‚1098.7939720153809 disk.nvme0n1.in‚0 disk.nvme0c0n1.in‚0 disk.sda.in‚0 disk.sda.out‚ 24.765625 disk.dm-1.in‚0$ disk.nvme0n1.out‚2145.5810546875 disk.drbd1777.out‚0 disk.rbd3.in‚0! disk.dm-0.out‚2145.5810546875Fÿ€’¼‚(:‡ ä„åÔÁ´‚Ž gpu.0.gpu‚37.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚91.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*…!½‚(:– ä„åÔµ”î disk.sda.in‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚2244.99267578125 network.sent‚ 1.009403e+06% disk./.usageGB‚1098.7939109802246 disk.dm-0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 25.3984375 disk.nvme0c0n1.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 25.3984375 disk.drbd1777.out‚0$ memory_percent‚97.71945476531982' disk.nvme0c0n1.out‚2244.99267578125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚37.625 disk.rbd3.in‚0 cpu‚25.037082381307002 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0) proc.memory.percent‚16.96462631225586 disk.sda.out‚ 25.3984375 disk.rbd3.out‚0! proc.memory.rssMB‚ 2779.484375" disk.dm-0.out‚2244.99267578125 disk.sda2.out‚0( proc.memory.availableMB‚ 373.64453125 disk.rbd1.out‚ 0.306640625 proc.cpu.threads‚17 network.recv‚295166) disk./.usagePercent‚73.74098131538229Q>Î0¾+ _wandbruntime‚599  _runtime‚599äÆّ¿¼ _runtime‚ 599.663295589 _step‚64 & lr-AdamW/pg1‚0.0001591420702243109 & lr-AdamW/pg2‚0.0001591420702243109 trainer/global_step‚849 # _timestamp‚1.78842891495856e+09’ƒÁÆÀÀ _runtime‚ 599.663295589 _step‚64 & lr-AdamW/pg1‚0.0001591420702243109 & lr-AdamW/pg2‚0.0001591420702243109 trainer/global_step‚849 # _timestamp‚1.78842891495856e+09@ê³0Á+ _wandbruntime‚599  _runtime‚599ð‹b’º'´O XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47378814515170 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48614159260160 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 696508157952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11921196 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 693948681186 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27677768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11921196 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚849 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10875946 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6948500 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47378143045602 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 169456640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 % _timestamp‚1.7884289149592967e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47916977503744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3927876 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52423758385664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15738030 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48614159260160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10876378 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15738030 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚65 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48613485661696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27659226 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 694603111394 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47916995506688 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53156626012672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 145516544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚ 599.664260087 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11921198 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47916995506688 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 696508157952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52423782325760 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53156795469312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48613485661696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27659226 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 694603111394 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47916995506688 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53156626012672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 145516544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚ 599.664260087 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11921198 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47916995506688 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 696508157952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52423782325760 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53156795469312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0A'Ûœ’Ä‚(:‡ ó„åÔØþÅ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚92.434' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ší9&Å‚(:› ó„åÔÀäÖ‘ disk.rbd4.out‚0 disk.dm-0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd4.in‚0 disk.sda.out‚ 25.9921875 disk.dm-1.out‚ 25.9921875 disk.nvme0n1.in‚0' disk.nvme0c0n1.out‚2245.30712890625 disk.rbd0.out‚0 network.sent‚ 1.015864e+06 disk.rbd2.out‚ 39.37109375 disk.nvme0c0n1.in‚0! proc.memory.rssMB‚ 2779.484375 network.recv‚299711 disk.sda1.out‚0% disk.nvme0n1.out‚2245.30712890625 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.037723926456458) disk./.usagePercent‚73.74098182739735 disk.rbd1.in‚0$ memory_percent‚97.72093296051025) proc.memory.percent‚16.96462631225586 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 25.9921875% disk./.usageGB‚1098.7939186096191 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0" disk.dm-0.out‚2245.30712890625 disk.sda.in‚0 disk.rbd1.out‚ 0.306640625( proc.memory.availableMB‚ 373.40234375 disk.rbd3.in‚0 disk.rbd3.out‚0îy#0Æ+ _wandbruntime‚600  _runtime‚600Tñ±•ÌǽM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27710264 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11935212 trainer/global_step‚849 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15775050 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3932516 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53219143973376 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 733897130496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10889186 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11935214 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 733734850048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11935212 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 _step‚66 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15775050 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 % _timestamp‚1.7884289156293936e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48671331740160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6956668 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47433863611126 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11935214 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47973330542592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15756484 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47973348545536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10888735 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 697327585792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52485409123328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27691696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48670658128384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46739098494976 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47973348545536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 697983194624 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47434535080902 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27710264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52485433063424 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 695419546566 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53219330193920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928ÁÔXqÇ&ÈÁM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27710264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52485433063424 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 695419546566 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53219330193920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27710264 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11935212 trainer/global_step‚849 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15775050 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3932516 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53219143973376 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 733897130496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10889186 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11935214 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 733734850048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11935212 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 _step‚66 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15775050 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 % _timestamp‚1.7884289156293936e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48671331740160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6956668 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47433863611126 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11935214 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47973330542592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15756484 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47973348545536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10888735 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 697327585792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52485409123328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27691696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48670658128384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46739098494976 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47973348545536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 697983194624 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47434535080902B> 0É+ _wandbruntime‚600  _runtime‚600ŒkkOÊÉ 2loss/SRqqt1/assignment_loss‚2.7312831542430778 /loss/FBt1/detection_loss‚1.0437149572706712 trainer/global_step‚849 _step‚67 /loss/FRt1/assignment_loss‚4.031803791587417 2loss/SRqqt2/assignment_loss‚3.1211236802543083 1loss/SRqqt4/assignment_loss‚2.318037678487599 /loss/FRt1/detection_loss‚1.9609985409958943 _runtime‚ 600.336198464 .loss/FRt2/assignment_loss‚5.98203175909379 /loss/FRt3/assignment_loss‚5.793442746426197 0loss/FBt3/assignment_loss‚0.5912612736428855 /loss/FRt3/detection_loss‚2.8843418106714145 /loss/SRqqt3/detection_loss‚2.62575179040432 /loss/FBt2/assignment_loss‚0.350100359210046 0loss/FBt4/assignment_loss‚0.4754184330185161 /loss/FBt2/detection_loss‚12.261508559187254 /loss/FBt4/detection_loss‚4.4443232623430395 /loss/FRt4/assignment_loss‚5.956989173231454 2loss/SRqqt3/assignment_loss‚2.7568171977996827 %loss/total_loss‚81.37541357820152 epoch‚0 0loss/FBt1/assignment_loss‚0.4094057180670916 /loss/FRt2/detection_loss‚3.6237912791616775 1loss/SRqqt1/detection_loss‚1.2663204179217036 1loss/SRqqt4/detection_loss‚3.6966888101564512 % _timestamp‚1.7884289156315546e+09 /loss/FRt4/detection_loss‚2.4585537581608214 0loss/SRqqt2/detection_loss‚2.895439198649511 ,loss/FBt3/detection_loss‚7.6962662282167êP&SËÍ 0loss/FBt3/assignment_loss‚0.5912612736428855 /loss/FRt3/detection_loss‚2.8843418106714145 /loss/SRqqt3/detection_loss‚2.62575179040432 /loss/FBt2/assignment_loss‚0.350100359210046 0loss/FBt4/assignment_loss‚0.4754184330185161 /loss/FBt2/detection_loss‚12.261508559187254 /loss/FBt4/detection_loss‚4.4443232623430395 /loss/FRt4/assignment_loss‚5.956989173231454 2loss/SRqqt3/assignment_loss‚2.7568171977996827 %loss/total_loss‚81.37541357820152 epoch‚0 0loss/FBt1/assignment_loss‚0.4094057180670916 /loss/FRt2/detection_loss‚3.6237912791616775 1loss/SRqqt1/detection_loss‚1.2663204179217036 1loss/SRqqt4/detection_loss‚3.6966888101564512 % _timestamp‚1.7884289156315546e+09 /loss/FRt4/detection_loss‚2.4585537581608214 0loss/SRqqt2/detection_loss‚2.895439198649511 ,loss/FBt3/detection_loss‚7.6962662282167 2loss/SRqqt1/assignment_loss‚2.7312831542430778 /loss/FBt1/detection_loss‚1.0437149572706712 trainer/global_step‚849 _step‚67 /loss/FRt1/assignment_loss‚4.031803791587417 2loss/SRqqt2/assignment_loss‚3.1211236802543083 1loss/SRqqt4/assignment_loss‚2.318037678487599 /loss/FRt1/detection_loss‚1.9609985409958943 _runtime‚ 600.336198464 .loss/FRt2/assignment_loss‚5.98203175909379 /loss/FRt3/assignment_loss‚5.793442746426197CEPHžÌ‚(:“ ‚…åÔ÷úžŽ gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚141.452' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.290399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹/WÂ(Í‚(: ‚…åÔ°×å disk.rbd0.in‚0 cpu‚25.037774108513197 network.sent‚ 1.053638e+06 disk.dm-0.in‚0! disk.dm-0.out‚2344.6884765625 disk.dm-1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 26.41796875 disk.nvme0n1.in‚0 proc.cpu.threads‚17 disk.sda.in‚0$ disk.nvme0n1.out‚2344.6884765625 disk.rbd1.out‚ 0.306640625 disk.rbd4.out‚0) disk./.usagePercent‚73.74100230800008 disk.rbd2.out‚ 39.37109375 disk.sda3.out‚ 26.41796875 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.sda.out‚ 26.41796875 disk.sda1.in‚0$ memory_percent‚97.71950244903564( proc.memory.availableMB‚ 373.63671875# proc.memory.rssMB‚ 2779.48828125 network.recv‚304532 disk.drbd1777.out‚0 disk.sda1.out‚0) proc.memory.percent‚16.96465015411377 disk.rbd3.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚2344.6884765625 disk.rbd1.in‚0 disk.rbd4.in‚0% disk./.usageGB‚1098.7942237854004 disk.sda3.in‚0À›Óz’΂(:‡ ‘…åÔÄÆî gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z2‡™"Ï‚(:— ‘…åÔê²ì disk.rbd3.in‚0$ memory_percent‚97.72119522094727 disk.sda2.out‚0 disk.dm-0.in‚0 disk.sda.in‚0 disk.sda.out‚ 26.9453125 disk.dm-1.in‚0 disk.dm-1.out‚ 26.9453125 disk.rbd1.out‚ 0.306640625 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚2443.7236328125 disk.sda3.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 proc.cpu.threads‚17 network.sent‚ 1.093017e+06 network.recv‚310060) disk./.usagePercent‚73.74100000393227! disk.dm-0.out‚2443.7236328125 disk.sda1.out‚0$ disk.nvme0n1.out‚2443.7236328125 disk.rbd0.in‚0 disk.rbd2.out‚ 41.11328125 disk.rbd3.out‚0& proc.memory.availableMB‚ 373.359375# proc.memory.rssMB‚ 2779.48828125) proc.memory.percent‚16.96465015411377 cpu‚25.038351077328308 disk.rbd1.in‚0 disk.drbd1777.out‚0$ disk./.usageGB‚1098.794189453125 disk.nvme0n1.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 26.9453125 disk.drbd1777.in‚0:×Ä0Ð+ _wandbruntime‚634  _runtime‚634(EÈÄѾ & lr-AdamW/pg1‚0.0001685143947369323 & lr-AdamW/pg2‚0.0001685143947369323 trainer/global_step‚899 % _timestamp‚1.7884289502435098e+09 _runtime‚ 634.948275139 _step‚68K"¥îÈÒ & lr-AdamW/pg1‚0.0001685143947369323 & lr-AdamW/pg2‚0.0001685143947369323 trainer/global_step‚899 % _timestamp‚1.7884289502435098e+09 _runtime‚ 634.948275139 _step‚68D¹'\0Ó+ _wandbruntime‚634  _runtime‚634M¿Kº'Ô´O C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49429417931776 _runtime‚ 634.949266927 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51472123409920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50164856382578 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 735438450802 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 737493963776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29304620 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16664082 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56282386057728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12621996 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚50734629446144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11517179 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚899 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 776253082624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12621996 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 776090788864 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29286078 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12621998 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 737493963776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12621998 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4159801 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚50734629446144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚50164184913010 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49429400892416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7356900 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16682622 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 734784020594 _step‚69 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51472123409920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 162293760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16664082 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928ãÈçÕ¸O QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11517179 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚899 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 776253082624 TFDeviceStatsMonitor.on_train_batch_start/rexs÷j×!served_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12621996 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 776090788864 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29286078 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12621998 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 737493963776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12621998 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4159801 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚50734629446144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚50164184913010 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49429400892416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7356900 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16682622 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 734784020594 _step‚69 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51472123409920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 162293760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16664082 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49429417931776 _runtime‚ 634.949266927 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51472123409920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50164856382578 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 735438450802 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 737493963776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29304620 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16664082 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56282386057728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12621996 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚50734629446144EÇ·UÈ0Ö+ _wandbruntime‚635  _runtime‚635ž‡¡šÄ&×¾M FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 738969000448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 776970960384 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50220576948310 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 736254885974 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 738969000448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12636012 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚899 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16682536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 738313391616 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12636014 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 777122755072 _step‚70 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884289509073653e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4164457 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 175734784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29337116 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚469 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11529525 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49484322062336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11529994 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29318548 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56345092701696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16701102 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29337116 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416hÅ<òÈ&ØÂM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16682536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚50219905478534 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50790982484992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 738969000448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12636012 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚899 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16682536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 738313391616 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12636014 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 777122755072 _step‚70 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884289509073653e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4164457 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 175734784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29337116 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚469 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11529525 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49484322062336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11529994 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29318548 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56345092701696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16701102 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29337116 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 738969000448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 776970960384 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50220576948310 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 736254885974 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54330761158144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172779520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _step‚73 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13322796 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17608674 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 776273790210 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17590134 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779135365120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54331434756608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7765302 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 775619360002 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4391682 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196719616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52174607420416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844ξ~´½'å·O MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚ 670.282467019 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54331434756608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7765302 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 775619360002 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4391682 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196719616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52174607420416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30912930 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53552281388544 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779135365120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30931472 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13322798 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 778479769600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚497 trainer/global_step‚949 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53552299391488 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52950898249986 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12156485 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52950226780418 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17590134 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4391185 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 819394773504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13322796 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788428985577478e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13322798 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53552299391488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58588832152064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54330761158144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172779520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _step‚73 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13322796 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17608674 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 776273790210 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17590134 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779135365120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54388607236608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4395730 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53608634427392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17627154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13336814 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52229511550976 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779299197440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53608652430336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779299197440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 779954806272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58650482889728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚477 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17627154 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53608652430336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53006618815718 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 777090225382 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚479 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 820091906048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4396207 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59470574795776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58650506829824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 779954806272 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53608634427392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30963968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13336812 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59470756822016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17608588 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884289862502167e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54388607236608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30945400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12169198 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54388607236608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4395730 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53608634427392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17627154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13336814 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52229511550976 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779299197440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53608652430336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779299197440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 779954806272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58650482889728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚477 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17627154 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53608652430336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53006618815718 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 777090225382 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚479 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 820091906048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4396207 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59470574795776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58650506829824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 779954806272 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53608634427392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30963968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13336812 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59470756822016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17608588 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884289862502167e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54388607236608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30945400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12169198 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EÆóÀ _runtime‚ 705.551145424 _step‚76 ' lr-AdamW/pg1‚0.00018725904376217504 ' lr-AdamW/pg2‚0.00018725904376217504 trainer/global_step‚999 % _timestamp‚1.7884290208463714e+09××uÊôÄ % _timestamp‚1.7884290208463714e+09 _runtime‚ 705.551145424 _step‚76 ' lr-AdamW/pg1‚0.00018725904376217504 ' lr-AdamW/pg2‚0.00018725904376217504 trainer/global_step‚999Lâ +0õ+ _wandbruntime‚705  _runtime‚705¦P'ö´O [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56369933330944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 862397734912 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54919813948416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56369933330944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18534726 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54919830987776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57190072504832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32558324 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 820121170944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56369951333888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8173702 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55736268647826 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12798119 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚62533961392640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14023598 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14023598 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61671392975360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61671369035264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 _step‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8173700 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18534726 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55736940117394 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14023596 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56369951333888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18516186 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 819465575424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4623949 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62533766770176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 705.552155668 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚999 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 817109129618 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57189398906368 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 862568417280 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884290208471925e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12797649 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 819465575424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4624417 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 816454699410 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928"Àö*O÷¸O XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 819465575424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4624417 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 816454699410 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56369933330944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 862397734912 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54919813948416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56369933330944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18534726 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54919830987776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57190072504832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32558324 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 820121170944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56369951333888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8173702 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55736268647826 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12798119 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚62533961392640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14023598 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14023598 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61671392975360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61671369035264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 _step‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8173700 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18534726 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55736940117394 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14023596 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56369951333888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18516186 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 819465575424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4623949 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62533766770176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 705.552155668 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚999 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 817109129618 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57189398906368 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 862568417280 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884290208471925e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12797649 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Më¯û0ø+ _wandbruntime‚706  _runtime‚706‹7¾=Ã&ù½M UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4629022 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57247244984832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61733043713024 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56426286369792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12810892 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788429021520555e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 863415722496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55791989213350 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32590820 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4628549 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62596260632064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 820940612096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32572252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18534640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56426304372736 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 820940612096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚ 706.225873076 _step‚78 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32572252 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198803456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57246571373056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18553206 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55792660683126 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8181868 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57247244984832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8181870 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174863360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 817925564790 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18534640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62596459435520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32590820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14037612 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56426304372736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56426286369792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 863240859136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12810417ÅrÉÇ&úÁM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57247244984832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8181870 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174863360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 817925564790 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18534640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62596459435520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32590820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14037612 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56426304372736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56426286369792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 863240859136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12810417 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4629022 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57247244984832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61733043713024 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56426286369792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12810892 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788429021520555e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 863415722496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55791989213350 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32590820 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4628549 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62596260632064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 820940612096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32572252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18534640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56426304372736 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 820940612096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚ 706.225873076 _step‚78 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32572252 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198803456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57246571373056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18553206 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55792660683126 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8181868 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903ND²@0û+ _wandbruntime‚706  _runtime‚706{1”OüÉ 1loss/SRqqt4/assignment_loss‚1.958861762975774 0loss/FBt1/assignment_loss‚0.2999916372405601 1loss/SRqqt1/detection_loss‚1.3099470961539608 _step‚79 0loss/FBt4/assignment_loss‚0.5059113069647603 /loss/FRt3/assignment_loss‚5.691431068792576 0loss/FBt2/assignment_loss‚0.1345781840616837 epoch‚0 .loss/FRt2/detection_loss‚2.419484400878782 0loss/FBt3/assignment_loss‚0.7819645266916434 /loss/SRqqt2/detection_loss‚2.07351209913139 /loss/FRt3/detection_loss‚2.6977974230923305 .loss/FBt1/detection_loss‚1.262394069110934 trainer/global_step‚999 0loss/FRt1/assignment_loss‚3.6906604585001026 /loss/FRt4/assignment_loss‚4.852410713831584 2loss/SRqqt1/assignment_loss‚2.5741768189838954 2loss/SRqqt2/assignment_loss‚2.3038839311511428 1loss/SRqqt4/detection_loss‚2.8920491148785845 _runtime‚ 706.227219102 /loss/FRt4/detection_loss‚2.9606118826639083 1loss/SRqqt3/detection_loss‚3.0354381408542395 .loss/FBt3/detection_loss‚6.858315852555362 -loss/FBt4/detection_loss‚4.47868747938247 % _timestamp‚1.7884290215225654e+09 /loss/FRt1/detection_loss‚2.3617778525261555 /loss/FRt2/assignment_loss‚5.141101940818455 1loss/SRqqt3/assignment_loss‚2.849277578294277 .loss/FBt2/detection_loss‚7.033118510246277 %loss/total_loss‚70.16738384978085$ÿlSýÍ 0loss/FRt1/assignment_loss‚3.6906604585001026 /loss/FRt4/assignment_loss‚4.852410713831584 2loss/SRqqt1/assignment_loss‚2.5741768189838954 2loss/SRqqt2/assignment_loss‚2.3038839311511428 1loss/SRqqt4/detection_loss‚2.8920491148785845 _runtime‚ 706.227219102 /loss/FRt4/detection_loss‚2.9606118826639083 1loss/SRqqt3/detection_loss‚3.0354381408542395 .loss/FBt3/detection_loss‚6.858315852555362 -loss/FBt4/detection_loss‚4.47868747938247 % _timestamp‚1.7884290215225654e+09 /loss/FRt1/detection_loss‚2.3617778525261555 /loss/FRt2/assignment_loss‚5.141101940818455 1loss/SRqqt3/assignment_loss‚2.849277578294277 .loss/FBt2/detection_loss‚7.033118510246277 %loss/total_loss‚70.16738384978085 1loss/SRqqt4/assignment_loss‚1.958861762975774 0loss/FBt1/assignment_loss‚0.2999916372405601 1loss/SRqqt1/detection_loss‚1.3099470961539608 _step‚79 0loss/FBt4/assignment_loss‚0.5059113069647603 /loss/FRt3/assignment_loss‚5.691431068792576 0loss/FBt2/assignment_loss‚0.1345781840616837 epoch‚0 .loss/FRt2/detection_loss‚2.419484400878782 0loss/FBt3/assignment_loss‚0.7819645266916434 /loss/SRqqt2/detection_loss‚2.07351209913139 /loss/FRt3/detection_loss‚2.6977974230923305 .loss/FBt1/detection_loss‚1.262394069110934 trainer/global_step‚999OZšáXþ‚(:‘ ë…åÔ§ÊÅ disk.dm-1.out‚ 30.51953125 proc.cpu.threads‚17 network.recv‚337885 disk.sda2.in‚0 disk.rbd4.out‚0# proc.memory.rssMB‚ 2779.48828125 disk.sda1.in‚0$ disk./.usageGB‚1098.794174194336$ disk.nvme0c0n1.out‚ 2842.29296875 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.dm-0.out‚ 2842.29296875 disk.nvme0n1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0$ memory_percent‚97.72095680236816) proc.memory.percent‚16.96465015411377) disk./.usagePercent‚73.74099897990214 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625 disk.nvme0c0n1.in‚0" disk.nvme0n1.out‚ 2842.29296875' proc.memory.availableMB‚ 373.3984375 cpu‚25.03977730805178 disk.rbd2.in‚0 disk.rbd2.out‚46.28125 disk.sda.in‚0 disk.sda.out‚ 30.51953125 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 network.sent‚ 1.215923e+06 disk.sda3.out‚ 30.51953125 disk.dm-1.in‚0 disk.dm-0.in‚0šxqŽÿ‚(:… ë…åÔþƒï¤ gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚93.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑÀ¤R“€‚(:ˆ ú…åÔ– Ý gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.292' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.4584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û:„´$‚(:™ ú…åÔ£Ã& proc.memory.availableMB‚ 373.640625 network.recv‚342782 disk.dm-1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 network.sent‚ 1.264726e+06 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 31.01953125 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2779.4921875 cpu‚25.039651733177955 disk.nvme0n1.in‚0 disk.sda3.out‚ 31.01953125 disk.rbd4.in‚0! disk.dm-0.out‚2941.7607421875 disk.sda.in‚0$ memory_percent‚97.71947860717773) disk./.usagePercent‚73.74098848359324$ disk./.usageGB‚1098.794017791748 disk.rbd1.in‚0& disk.nvme0c0n1.out‚2941.7607421875 disk.rbd2.in‚0 disk.rbd2.out‚ 47.85546875 disk.drbd1777.in‚0 disk.sda.out‚ 31.01953125 disk.rbd1.out‚ 0.306640625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17$ disk.nvme0n1.out‚2941.7607421875 disk.drbd1777.out‚0 disk.rbd4.out‚0) proc.memory.percent‚16.96467399597168 disk.dm-0.in‚0Æ¢²0‚+ _wandbruntime‚741  _runtime‚741Äô–Îý _runtime‚ 741.046886189 _step‚80 & lr-AdamW/pg1‚0.0001966313682747964 & lr-AdamW/pg2‚0.0001966313682747964 trainer/global_step‚1049 # _timestamp‚1.78842905634214e+09åŒZ>Ç„Á & lr-AdamW/pg1‚0.0001966313682747964 & lr-AdamW/pg2‚0.0001966313682747964 trainer/global_step‚1049 # _timestamp‚1.78842905634214e+09 _runtime‚ 741.046886189 _step‚80P¼,¼w0…+ _wandbruntime‚741  _runtime‚741{]ì†H†µO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14724396 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13438317 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65659533895680 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60048036654592 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59187603276288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4856215 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64753929858560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14724398 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 861106976768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65659722226688 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19460778 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4855754 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34166634 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 905792368128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 860451381248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 861106976768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13437854 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8582100 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59187585273344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59187603276288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64753905918464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605ñ”õÊ¿'‡¹O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 857944469026 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65659533895680 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60048036654592 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59187603276288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4856215 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64753929858560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14724398 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 861106976768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65659722226688 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19460778 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4855754 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34166634 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 905792368128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 860451381248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 861106976768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13437854 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8582100 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59187585273344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59187603276288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64753905918464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14724396 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13438317 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58578702550534 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19479258 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59243956315136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34217672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64815556656128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14738414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8590270 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59243938312192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 858106473782 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884290570058453e+09 _step‚82 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60105209121280 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58578031080758 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 858760904198 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 861270809088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 906495420416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19479258 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4860466 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19460692 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚1049 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13450734 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13451188 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60105882733056 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57719941646336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928´xØ=¶ŠÃM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19479258 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59243956315136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34217672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64815556656128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14738414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8590270 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59243938312192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 858106473782 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884290570058453e+09 _step‚82 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60105209121280 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58578031080758 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 858760904198 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 861270809088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 906495420416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19479258 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4860466 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19460692 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚1049 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13450734 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13451188 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60105882733056 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57719941646336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59243938312192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14738414 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65722052076544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14738412 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64815580596224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19460692 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60105882733056 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59243956315136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58578702550534 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704R Fñ&0‹+ _wandbruntime‚741  _runtime‚741¡ŠúÆVŒÐ /loss/FBt3/assignment_loss‚1.070093184709549 1loss/SRqqt1/detection_loss‚0.8336503226039281 0loss/SRqqt4/detection_loss‚2.452797614715316 /loss/FBt4/detection_loss‚3.9935538579117167 epoch‚0 trainer/global_step‚1049 % _timestamp‚1.7884290570078285e+09 /loss/FRt4/assignment_loss‚4.652246452146961 2loss/SRqqt3/assignment_loss‚2.7377422131024876 _runtime‚ 741.712527311 /loss/FRt2/assignment_loss‚4.932575274736453 2loss/SRqqt4/assignment_loss‚2.1046820045872168 1loss/FBt1/assignment_loss‚0.24431370869986554 /loss/FBt4/assignment_loss‚0.453531643608585 /loss/FRt2/detection_loss‚2.7145077349283757 %loss/total_loss‚65.40893328104924 _step‚83 2loss/SRqqt1/assignment_loss‚2.4546060451241427 2loss/SRqqt2/assignment_loss‚2.7888916647711466 0loss/FRt1/assignment_loss‚3.0649686253839925 1loss/FBt2/assignment_loss‚0.24497604618469873 1loss/SRqqt2/detection_loss‚2.4885316678257876 /loss/FBt1/detection_loss‚1.4463918697530327 .loss/FRt1/detection_loss‚2.094486347129268 /loss/FRt3/assignment_loss‚5.913578019823347 .loss/FRt3/detection_loss‚2.977442327035325 /loss/FRt4/detection_loss‚2.7234747409820557 1loss/SRqqt3/detection_loss‚3.2707614921606503 .loss/FBt2/detection_loss‚8.437376419703165 /loss/FBt3/detection_loss‚1.3137540034221848)zZÔ 0loss/FRt1/assignment_loss‚3.0649686253839925 1loss/FBt2/assignment_loss‚0.24497604618469873 1loss/SRqqt2/detection_loss‚2.4885316678257876 /loss/FBt1/detection_loss‚1.4463918697530327 .loss/FRt1/detection_loss‚2.094486347129268 /loss/FRt3/assignment_loss‚5.913578019823347 .loss/FRt3/detection_loss‚2.977442327035325 /loss/FRt4/detection_loss‚2.7234747409820557 1loss/SRqqt3/detection_loss‚3.2707614921606503 .loss/FBt2/detection_loss‚8.437376419703165 /loss/FBt3/detection_loss‚1.3137540034221848 /loss/FBt3/assignment_loss‚1.070093184709549 1loss/SRqqt1/detection_loss‚0.8336503226039281 0loss/SRqqt4/detection_loss‚2.452797614715316 /loss/FBt4/detection_loss‚3.9935538579117167 epoch‚0 trainer/global_step‚1049 % _timestamp‚1.7884290570078285e+09 /loss/FRt4/assignment_loss‚4.652246452146961 2loss/SRqqt3/assignment_loss‚2.7377422131024876 _runtime‚ 741.712527311 /loss/FRt2/assignment_loss‚4.932575274736453 2loss/SRqqt4/assignment_loss‚2.1046820045872168 1loss/FBt1/assignment_loss‚0.24431370869986554 /loss/FBt4/assignment_loss‚0.453531643608585 /loss/FRt2/detection_loss‚2.7145077349283757 %loss/total_loss‚65.40893328104924 _step‚83 2loss/SRqqt1/assignment_loss‚2.4546060451241427 2loss/SRqqt2/assignment_loss‚2.7888916647711466SõqŽß’Ž‚(:‡ ‰†åÔô“ gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.715' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.943 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Oe."‚(:— ‰†åÔÆ¢( proc.memory.availableMB‚ 373.05078125" proc.memory.rssMB‚ 2779.4921875 disk.rbd1.out‚ 0.455078125 disk.rbd2.out‚ 48.0234375 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.rbd4.out‚0 disk.sda.out‚ 31.62109375 network.recv‚347312# disk.nvme0n1.out‚3040.892578125 disk.drbd1777.out‚0 disk.dm-0.out‚3040.892578125 disk.sda.in‚0 disk.dm-1.out‚ 31.62109375$ memory_percent‚97.72307872772217% disk.nvme0c0n1.out‚3040.892578125 disk.sda2.out‚0 network.sent‚ 1.271037e+06( disk./.usagePercent‚73.7409889956083 disk.rbd4.in‚0 disk.sda1.in‚0) proc.memory.percent‚16.96467399597168 cpu‚25.040191008258056 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1098.7940254211426 disk.nvme0c0n1.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 31.62109375 disk.sda1.out‚0 disk.dm-1.in‚0dñ¯ž‚(:“ ˜†åÔ„Ì gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.759' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.551799999999993 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßdà}!‘‚(:– ˜†åÔüçô disk.rbd2.out‚ 49.53515625 disk.sda1.in‚0% disk.nvme0n1.out‚3140.60693359375) proc.memory.percent‚16.96467399597168 network.recv‚352233 disk.dm-0.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0 cpu‚25.04033553987514 network.sent‚ 1.320066e+06) disk./.usagePercent‚73.74102150856515' disk.nvme0c0n1.out‚3140.60693359375 disk.sda3.in‚0$ memory_percent‚97.72138595581055% disk./.usageGB‚1098.7945098876953 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0" disk.dm-0.out‚3140.60693359375 disk.sda1.out‚0 disk.dm-1.out‚ 32.234375 disk.sda3.out‚ 32.234375 disk.nvme0c0n1.in‚0 disk.drbd1777.in‚0 disk.sda.out‚ 32.234375 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0" proc.memory.rssMB‚ 2779.4921875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.455078125& proc.memory.availableMB‚ 373.328125 disk.rbd0.out‚0 disk.rbd3.out‚02³YÃ0’+ _wandbruntime‚776  _runtime‚776ß81£Ç“Á _runtime‚ 776.433346296 _step‚84 ' lr-AdamW/pg1‚0.00020600369278741778 ' lr-AdamW/pg2‚0.00020600369278741778 trainer/global_step‚1099 % _timestamp‚1.7884290917285924e+09Ç0'˔Š' lr-AdamW/pg1‚0.00020600369278741778 ' lr-AdamW/pg2‚0.00020600369278741778 trainer/global_step‚1099 % _timestamp‚1.7884290917285924e+09 _runtime‚ 776.433346296 _step‚84TH…G0•+ _wandbruntime‚776  _runtime‚776í-Ë»'–µO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20386830 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62005255218688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 898779808434 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35812028 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15425198 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60410227004416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62906674402816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62005237215744 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚477 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14078655 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20368290 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15425196 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 902092782592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15425196 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60410244043776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 902092782592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8990500 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8990502 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15425198 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62005255218688 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414^šþ/¿'—¹O dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67836466741760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5088153 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68785499517952 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61309023852210 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62907348001280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15425198 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62005255218688 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20386830 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62005255218688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 898779808434 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35812028 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15425198 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60410227004416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62906674402816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62005237215744 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚477 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14078655 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20368290 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15425196 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 902092782592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15425196 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60410244043776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 902092782592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8990500 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8990502 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚62061608257536 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35844524 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5092353 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8998668 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60465131134976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62061608257536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15439214 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20386744 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67898093539328 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8998670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 902912223744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68848013410304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 949737844736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61364072948166 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 898941813190 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62964520481280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 902256614912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15439212 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15439214 trainer/global_step‚1099 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67898117479424 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 $ _timestamp‚1.788429092395862e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62964520481280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14091479 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14091021 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 899596243606 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20405310 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61364744417942 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 949895930880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚86 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35844524 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 902256614912 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5092809 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚62061608257536 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35844524 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5092353 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8998668 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60465131134976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62061608257536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15439214 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20386744 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67898093539328 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8998670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 902912223744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68848013410304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 949737844736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61364072948166 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 898941813190 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62964520481280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 902256614912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15439212 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15439214 trainer/global_step‚1099 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67898117479424 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 $ _timestamp‚1.788429092395862e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62964520481280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14091479 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14091021 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 899596243606V3_ï÷0›+ _wandbruntime‚777  _runtime‚777`OúôVœÐ $loss/total_loss‚65.4551396269134 1loss/FBt1/assignment_loss‚0.24542665538424852 epoch‚0 _runtime‚ 777.102185588 1loss/SRqqt1/detection_loss‚1.3642553986266388 .loss/FBt3/detection_loss‚6.014348763227463 .loss/FRt1/detection_loss‚2.203426854809125 /loss/FRt3/assignment_loss‚5.784000780524277 2loss/SRqqt2/assignment_loss‚2.4927615536020156 /loss/FRt3/detection_loss‚2.6470765667944782 /loss/FRt4/detection_loss‚2.5990109750202723 0loss/FRt1/assignment_loss‚2.5470170855522154 1loss/SRqqt3/detection_loss‚2.9072315759128995 .loss/FBt2/detection_loss‚5.971270203590393 trainer/global_step‚1099 1loss/SRqqt3/assignment_loss‚2.913445508038556 1loss/SRqqt4/assignment_loss‚2.663658321105145 /loss/FBt4/assignment_loss‚0.449092081407746 /loss/FBt1/detection_loss‚0.7490497682278782 0loss/FRt2/assignment_loss‚4.5037315258612995 /loss/FRt4/assignment_loss‚5.332169110434396 1loss/SRqqt2/detection_loss‚2.1489186581779034 % _timestamp‚1.7884290923975732e+09 2loss/SRqqt1/assignment_loss‚2.4069454410246442 0loss/FBt2/assignment_loss‚0.7559440974146128 1loss/FBt3/assignment_loss‚0.26063075006441067 _step‚87 /loss/FRt2/detection_loss‚2.3551488696382594 1loss/SRqqt4/detection_loss‚2.6463099871132827 /loss/FBt4/detection_loss‚3.4942690953612328³¶MZÔ 1loss/SRqqt4/assignment_loss‚2.663658321105145 /loss/FBt4/assignment_loss‚0.449092081407746 /loss/FBt1/detection_loss‚0.7490497682278782 0loss/FRt2/assignment_loss‚4.5037315258612995 /loss/FRt4/assignment_loss‚5.332169110434396 1loss/SRqqt2/detection_loss‚2.1489186581779034 % _timestamp‚1.7884290923975732e+09 2loss/SRqqt1/assignment_loss‚2.4069454410246442 0loss/FBt2/assignment_loss‚0.7559440974146128 1loss/FBt3/assignment_loss‚0.26063075006441067 _step‚87 /loss/FRt2/detection_loss‚2.3551488696382594 1loss/SRqqt4/detection_loss‚2.6463099871132827 /loss/FBt4/detection_loss‚3.4942690953612328 $loss/total_loss‚65.4551396269134 1loss/FBt1/assignment_loss‚0.24542665538424852 epoch‚0 _runtime‚ 777.102185588 1loss/SRqqt1/detection_loss‚1.3642553986266388 .loss/FBt3/detection_loss‚6.014348763227463 .loss/FRt1/detection_loss‚2.203426854809125 /loss/FRt3/assignment_loss‚5.784000780524277 2loss/SRqqt2/assignment_loss‚2.4927615536020156 /loss/FRt3/detection_loss‚2.6470765667944782 /loss/FRt4/detection_loss‚2.5990109750202723 0loss/FRt1/assignment_loss‚2.5470170855522154 1loss/SRqqt3/detection_loss‚2.9072315759128995 .loss/FBt2/detection_loss‚5.971270203590393 trainer/global_step‚1099 1loss/SRqqt3/assignment_loss‚2.913445508038556WÈ·@‘ž‚(:† §†åÔžŠ—Ž gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.87' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ZÅ+Ÿ‚(:  §†åÔÔ…¢‘ disk.rbd3.in‚0 disk.nvme0n1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.dm-1.out‚ 32.90234375 disk.rbd1.in‚0 disk.rbd1.out‚ 0.4873046875& disk.nvme0c0n1.out‚3239.7763671875 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd2.out‚49.64892578125 disk.drbd1777.in‚0! disk.dm-0.out‚3239.7763671875( proc.memory.availableMB‚ 373.05859375 network.sent‚ 1.326322e+06$ disk.nvme0n1.out‚3239.7763671875 disk.sda3.in‚0$ memory_percent‚97.72303104400635" proc.memory.rssMB‚ 2779.4921875 network.recv‚356664) disk./.usagePercent‚73.74102586069323 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0% disk./.usageGB‚1098.7945747375488 disk.sda.in‚0 disk.sda.out‚ 32.90234375 disk.sda2.out‚0 disk.sda3.out‚ 32.90234375 disk.dm-0.in‚0) proc.memory.percent‚16.96467399597168 cpu‚25.040491994920906 proc.cpu.threads‚17ï bh“ ‚(:ˆ ¶†åÔ¼¬‚Ž gpu.0.gpu‚41.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚158.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.7524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´Gá$¡‚(:™ ¶†åÔƒóô‘ network.sent‚ 1.375524e+06! disk.dm-0.out‚3240.9267578125 disk.dm-1.out‚ 33.67578125 disk.sda3.out‚ 33.67578125 disk.rbd3.in‚0$ memory_percent‚97.72064685821533 cpu‚25.040624742463056 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 proc.memory.rssMB‚2779.5 proc.cpu.threads‚17) disk./.usagePercent‚73.74102970080625 disk.sda1.in‚0& disk.nvme0c0n1.out‚3240.9267578125 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0( proc.memory.availableMB‚ 373.44921875 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚51.16064453125( proc.memory.percent‚16.9647216796875% disk./.usageGB‚1098.7946319580078 disk.nvme0n1.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 33.67578125 disk.dm-1.in‚0 disk.sda3.in‚0 network.recv‚361815 disk.dm-0.in‚0 disk.rbd4.in‚0$ disk.nvme0n1.out‚3240.9267578125 disk.rbd1.out‚ 0.4873046875 disk.rbd4.out‚0Â8=F“¢‚(:ˆ ņåÔÃÕÝ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚147.121' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.4242 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ÖôÕ£‚(:“ ņåÔ¦¬ü disk.sda3.out‚ 34.234375 disk.drbd1777.in‚0 disk.rbd3.out‚0$ memory_percent‚97.72217273712158 network.recv‚366477! disk.dm-0.out‚3340.0869140625 disk.sda1.in‚0 disk.rbd1.out‚ 0.55029296875 cpu‚25.040614742741628 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.nvme0n1.in‚0 disk.sda2.out‚0 network.sent‚ 1.38196e+06 disk.drbd1777.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 34.234375 disk.rbd1.in‚0 proc.memory.rssMB‚2779.5) disk./.usagePercent‚73.74102739673843 disk.nvme0c0n1.in‚0 disk.sda2.in‚0% disk./.usageGB‚1098.7945976257324 disk.dm-0.in‚0& disk.nvme0c0n1.out‚3340.0869140625 disk.rbd0.out‚0( proc.memory.availableMB‚ 373.19921875 disk.sda3.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0( proc.memory.percent‚16.9647216796875 disk.rbd3.in‚0 disk.sda.out‚ 34.234375$ disk.nvme0n1.out‚3340.0869140625 disk.rbd2.out‚51.32470703125º™:M0¤+ _wandbruntime‚811  _runtime‚811ù õýÇ¥Á trainer/global_step‚1149 % _timestamp‚1.7884291272466946e+09 _runtime‚ 811.951436024 _step‚88 ' lr-AdamW/pg1‚0.00021537601730003915 ' lr-AdamW/pg2‚0.00021537601730003915þ" )˦Š% _timestamp‚1.7884291272466946e+09 _runtime‚ 811.951436024 _step‚88 ' lr-AdamW/pg1‚0.00021537601730003915 ' lr-AdamW/pg2‚0.00021537601730003915 trainer/global_step‚1149X„±ö0§+ _wandbruntime‚811  _runtime‚811}éÜœ¨µO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16125998 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70918979684864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚1149 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚89 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5320885 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65765312151040 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65765985749504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64822889158144 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 943078588416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 175748096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21312882 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64822889158144 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71911221374976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63155450571776 _runtime‚ 811.952389571 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37420338 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 943078588416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 942422992896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71911045626880 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16125996 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64822907161088 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14719787 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65765312151040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21312882 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9398900 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64095065719618 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64822907161088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37438880 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21294342 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 938960717634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16125996 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14719323 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 939615147842 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992217750016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 942422992896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5320423 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64094394250050 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65765312151040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21312882 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9398900 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64095065719618 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64822907161088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37438880 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21294342 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 938960717634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16125996 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14719323 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 939615147842 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992217750016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 942422992896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5320423 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64094394250050 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚89 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5320885 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65765312151040 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65765985749504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64822889158144 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 943078588416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 175748096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21312882 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64822889158144 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71911221374976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63155450571776 _runtime‚ 811.952389571 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37420338 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 943078588416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 942422992896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71911045626880Y$(&0ª+ _wandbruntime‚812  _runtime‚812±µ€Ä&«¾M TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37471376 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64150786285350 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64879242196992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65822484617728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21331362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14732087 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16140014 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16140012 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70980630422528 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 940431583014 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21331362 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37452808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63210354702336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63210337662976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21312796 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21312796 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64879260199936 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 943898029568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 _step‚90 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 943242420736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71973740571648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71973550156800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9407068 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16140012 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37471376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 992919734272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64150114815574 trainer/global_step‚1149 _runtime‚ 812.61892798 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 939777152598 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884291279134572e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65823158229504 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 943242420736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 943898029568 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16140014 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64879260199936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70980654362624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14732558 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64879242196992 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9407070 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993086209024 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14732087 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16140014 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16140012 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70980630422528 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 940431583014 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21331362 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37452808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63210354702336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63210337662976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21312796 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21312796 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64879260199936 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 943898029568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 _step‚90 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 943242420736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71973740571648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71973550156800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9407068 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16140012 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37471376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 992919734272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64150114815574 trainer/global_step‚1149 _runtime‚ 812.61892798 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 939777152598 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884291279134572e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65823158229504 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 943242420736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 943898029568 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16140014 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64879260199936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70980654362624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 [MDeviceStatsMonitor.on_train_batch_end/inactive_splˆË½it_bytes.small_pool.current‚ 166474752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14732558 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64879242196992 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9407070 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993086209024 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74001516568064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16826798 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1035310102528 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66880436117458 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚1199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16826796 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 983408798720 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75036662279680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65900657099776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68624623497728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68623949899264 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75036850610688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22238934 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9807302 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22220394 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚1199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16826796 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 983408798720 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75036662279680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65900657099776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68624623497728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68623949899264 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75036850610688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22238934 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9807302 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22220394 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74001516568064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16826798 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1035310102528 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66880436117458 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9815470 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22257414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 984228226560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75099364015616 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚75099177795072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884291631922514e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚67696894139392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16840812 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚67696912142336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 980612492006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9815468 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚67696912142336 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 984883835392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚478 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68681122365952 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39079660 B8DeviceStatsMonitor.on_trai€ëÌ@÷n_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65955544190976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22238848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15372574 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74063191245824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16840814 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1036172769792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65955561230336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66936156682982 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 _step‚94 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 981266922422 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39098228 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5557104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15372096 _runtime‚ 847.89761113 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1199 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74063167305728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1036010489344 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39098228 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22238848 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚67696894139392 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5556628 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9815470 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22257414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 984228226560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75099364015616 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚75099177795072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884291631922514e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚67696894139392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16840812 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚67696912142336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 980612492006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9815468 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚67696912142336 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 984883835392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚478 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68681122365952 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39079660 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65955544190976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22238848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15372574 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74063191245824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16840814 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1036172769792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65955561230336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66936156682982 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 _step‚94 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 981266922422 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39098228 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5557104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15372096 _runtime‚ 847.89761113 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1199 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74063167305728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1036010489344 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39098228 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22238848 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚67696894139392 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5556628 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544^æÚ0½+ _wandbruntime‚847  _runtime‚847¥Á°R¾Ì .loss/FBt2/detection_loss‚7.646937549114227 _step‚95 1loss/SRqqt3/assignment_loss‚2.732267924149831 /loss/FRt4/detection_loss‚2.4209425381638785 .loss/FBt3/detection_loss‚4.785867697662777 .loss/FBt4/detection_loss‚4.222866590533938 trainer/global_step‚1199 /loss/FRt2/assignment_loss‚4.570434798585608 /loss/FRt4/assignment_loss‚4.488516523079439 1loss/FBt3/assignment_loss‚0.41544015034337967 %loss/total_loss‚64.27849740109835 .loss/FRt2/detection_loss‚2.413750983933185 /loss/FRt1/assignment_loss‚2.922418516571239 1loss/SRqqt2/assignment_loss‚2.235090208715863 0loss/FBt2/assignment_loss‚0.1976061286404729 0loss/SRqqt3/detection_loss‚2.599141412973404 _runtime‚ 847.899013543 1loss/SRqqt4/assignment_loss‚2.560210437190776 /loss/FRt3/detection_loss‚2.9362759283094695 % _timestamp‚1.7884291631943846e+09 .loss/FRt1/detection_loss‚2.248915634165376 /loss/FRt3/assignment_loss‚5.102144956588745 2loss/SRqqt1/assignment_loss‚2.2791327238082886 1loss/FBt1/assignment_loss‚0.28062590169759966 1loss/FBt4/assignment_loss‚0.49275004437509196 1loss/SRqqt2/detection_loss‚2.4271559410625034 0loss/SRqqt4/detection_loss‚2.175022723723431 1loss/SRqqt1/detection_loss‚1.4631192408502103 /loss/FBt1/detection_loss‚0.6618628468596078 epoch‚0’ˆØRV¿Ð % _timestamp‚1.7884291631943846e+09 .loss/FRt1/detection_loss‚2.248915634165376 /loss/FRt3/assignment_loss‚5.102144956588745 2loss/SRqqt1/assignment_loss‚2.2791327238082886 1loss/FBt1/assignment_loss‚0.28062590169759966 1loss/FBt4/assignment_loss‚0.49275004437509196 1loss/SRqqt2/detection_loss‚2.4271559410625034 0loss/SRqqt4/detection_loss‚2.175022723723431 1loss/SRqqt1/detection_loss‚1.4631192408502103 /loss/FBt1/detection_loss‚0.6618628468596078 epoch‚0 .loss/FBt2/detection_loss‚7.646937549114227 _step‚95 1loss/SRqqt3/assignment_loss‚2.732267924149831 /loss/FRt4/detection_loss‚2.4209425381638785 .loss/FBt3/detection_loss‚4.785867697662777 .loss/FBt4/detection_loss‚4.222866590533938 trainer/global_step‚1199 /loss/FRt2/assignment_loss‚4.570434798585608 /loss/FRt4/assignment_loss‚4.488516523079439 1loss/FBt3/assignment_loss‚0.41544015034337967 %loss/total_loss‚64.27849740109835 .loss/FRt2/detection_loss‚2.413750983933185 /loss/FRt1/assignment_loss‚2.922418516571239 1loss/SRqqt2/assignment_loss‚2.235090208715863 0loss/FBt2/assignment_loss‚0.1976061286404729 0loss/SRqqt3/detection_loss‚2.599141412973404 _runtime‚ 847.899013543 1loss/SRqqt4/assignment_loss‚2.560210437190776 /loss/FRt3/detection_loss‚2.9362759283094695_žŒ€sÀ‚(:’ ò†åÔþȪŽ gpu.0.gpu‚44.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.695' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.538999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QòšJ-Á‚(:¢ ò†åÔ焇 disk.rbd0.in‚0# proc.memory.rssMB‚ 2855.36328125 network.sent‚ 1.441595e+06 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚51.50146484375" disk.dm-0.out‚3539.74072265625 disk.sda3.out‚ 35.87890625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚3539.74072265625 disk.sda3.in‚0 disk.sda.out‚ 35.87890625$ memory_percent‚98.18708896636963 cpu‚25.046480778121303 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 297.02734375* proc.memory.percent‚17.427754402160645 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.sda2.out‚0$ disk./.usageGB‚1098.794750213623 disk.sda1.in‚0% disk.nvme0n1.out‚3539.74072265625 network.recv‚379064) disk./.usagePercent‚73.74103763703981 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 35.87890625 disk.rbd1.out‚ 0.61328125}‘‚(:† ‡åÔ¢ÍÙ gpu.0.gpu‚3.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚92.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wv('Â(:œ ‡åÔ·»% disk./.usageGB‚1098.7946968078613 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚53.15380859375 disk.rbd4.out‚0 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 36.6796875 disk.nvme0c0n1.in‚0 disk.sda2.in‚0 network.recv‚384670 disk.drbd1777.out‚0 disk.sda.out‚ 36.6796875$ memory_percent‚98.19142818450928 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 0.61328125 network.sent‚ 1.491586e+06 disk.sda3.out‚ 36.6796875 disk.drbd1777.in‚0 proc.memory.rssMB‚ 2856.3125 disk.rbd3.in‚0" disk.dm-0.out‚3638.84619140625 disk.nvme0n1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚3638.84619140625 disk.sda2.out‚0* proc.memory.percent‚17.433547973632812 cpu‚25.130799900147355) disk./.usagePercent‚73.74103405293432 disk.dm-0.in‚0 disk.sda.in‚0 disk.sda1.in‚0% disk.nvme0n1.out‚3638.84619140625( proc.memory.availableMB‚ 296.31640625áðùœÄ‚(:‘ ‡åÔðñ¹ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚86.763' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.352600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 '5Ú,Å‚(:¡ ‡åÔáùí proc.cpu.threads‚17 network.sent‚ 1.497049e+06 disk.sda3.out‚ 37.33203125 disk.rbd3.in‚0 disk.nvme0c0n1.in‚0* proc.memory.percent‚17.435336112976074 network.recv‚388401) disk./.usagePercent‚73.74102355662544 disk.rbd2.in‚0 disk.dm-0.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 cpu‚25.2082450101324 disk.rbd3.out‚0 disk.rbd4.out‚0" disk.dm-0.out‚3738.10986328125 disk.sda.in‚0 disk.dm-1.out‚ 37.33203125 disk.nvme0n1.in‚0% disk.nvme0n1.out‚3738.10986328125 disk.drbd1777.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 295.96484375 disk.rbd2.out‚53.15380859375 disk.sda3.in‚0 disk.rbd4.in‚0% disk./.usageGB‚1098.7945404052734 disk.rbd1.out‚ 0.61328125# proc.memory.rssMB‚ 2856.60546875 disk.rbd0.out‚0 disk.sda.out‚ 37.33203125 disk.dm-1.in‚0' disk.nvme0c0n1.out‚3738.10986328125 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚98.19357395172119º’cœÆ‚(:‘ Ÿ‡åÔ–ë× gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚86.621' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.324199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…öÒ'Ç‚(:œ Ÿ‡åÔ¯‰Ï‘# proc.memory.availableMB‚296.375 network.sent‚ 1.503365e+06 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 37.95703125 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.dm-0.in‚0% disk.nvme0n1.out‚3837.16845703125 disk.rbd0.in‚0) proc.memory.percent‚17.43309497833252 network.recv‚393236" disk.dm-0.out‚3837.16845703125 disk.sda1.in‚0 disk.dm-1.out‚ 37.95703125 disk.rbd1.in‚0% disk./.usageGB‚1098.7944869995117 disk.nvme0n1.in‚0' disk.nvme0c0n1.out‚3837.16845703125 disk.sda2.out‚0 cpu‚25.28549667417504 disk.sda3.out‚ 37.95703125 disk.dm-1.in‚0 disk.rbd2.out‚53.33447265625# proc.memory.rssMB‚ 2856.23828125 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.in‚0$ memory_percent‚98.19107055664062) disk./.usagePercent‚73.74101997251995 disk.nvme0c0n1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd4.out‚0 disk.rbd1.out‚ 0.61328125V)ˆSqÈbl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP¦}(£rÉbm JDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current"trainer/global_step2:JPGçŒrÊbm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP¤Ý kËbf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JP.¾ZnÌbi FDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current"trainer/global_step2:JP–€«rÍbm JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPô­‹ÞuÎbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP½ëe\pÏbk HDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed"trainer/global_step2:JPúå*zÐbu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPcy3·yÑbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP>Ä–vkÒbf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JPÛŠtÓbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP¼á¯tÔbo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPU8.OdÕb_ DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JP/¡ènìbi FDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current"trainer/global_step2:JPçË=víbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP²þ=6xîbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPõ^+wvïbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPÉéF sðbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP†Hò„yñbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPnqŒÎwòbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP?N‰vqóbl IDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated"trainer/global_step2:JPB}V…tôbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPEJUiõbd ADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed"trainer/global_step2:JP+Õ+uöbp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPwWÏ|÷bw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP(“õ%jøbe BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JP ã:€oùbj GDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak"trainer/global_step2:JP¬4‘"oúbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JPÉïÚyûbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPe«jübe BDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak"trainer/global_step2:JP@*ncýb^ ;DeviceStatsMonitor.on_validation_batch_start/max_split_size"trainer/global_step2:JPPuž@kþbf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JPþ¾Ànÿbi FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed"trainer/global_step2:JPA"lÃl€bg DDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP$—¶ãqbl IDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated"trainer/global_step2:JPÇI¶bl‚bg DDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed"trainer/global_step2:JPR3oƒbj GDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak"trainer/global_step2:JPNô‰d„b_ DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JPL´ÿgžbb ?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections"trainer/global_step2:JP0’¹vŸbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPè3¢m bh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JPP€±n¡bi FDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current"trainer/global_step2:JP–ίm¢bh EDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed"trainer/global_step2:JPø4/¸n£bi FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPdp@(r¤bm JDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current"trainer/global_step2:JPåðï(l¥bg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JPƒ…bÞq¦bl IDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current"trainer/global_step2:JPvù¹Tm§bh EDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPe¥Q÷x¨bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP«×pbt©bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPµ²¿Ÿxªbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPî/k«bf CDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak"trainer/global_step2:JP ¤”p¬bk HDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current"trainer/global_step2:JPäu²@s­bn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JP©¥ïÈh®bc @DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak"trainer/global_step2:JP n|y¯bt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPåèVO~°by VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP“.Wr±bm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPr“v²bq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPaÿÕdy³bt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP’iŽx´bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP»„cÃ]µbX 5DeviceStatsMonitor.on_validation_batch_start/num_ooms"trainer/global_step2:JP¤»°l¶bg DDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed"trainer/global_step2:JPüùL—v·bq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP_·£s¸bn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP?Úe¹b` =DeviceStatsMonitor.on_validation_batch_start/segment.all.peak"trainer/global_step2:JPLT¢­mºbh EDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed"trainer/global_step2:JP©8!p»bk HDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed"trainer/global_step2:JP¹­{t¼bo LDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP5-Tu½bp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPh>Í¢l¾bg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JP³Èôèr¿bm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPÓ¸SpÀbk HDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated"trainer/global_step2:JPã ÜsÁbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPy¤bnÂbi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JP¥Ç,0Ã+ _wandbruntime‚901  _runtime‚901o)ê¨,*ĦT PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15191 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚70008760903168 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚10207678 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 1006447712518 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490911116 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚40262510 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 135492096 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚156 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚68179092620288 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9954819072 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚69185032382330 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚16039405 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚17496450 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚22630 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚16039249 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚71018415323648 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15191 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚1208 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚5831727 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510430720 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚22903 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 1062021939712 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚70008742900224 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1010164851200 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚70008760903168 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚17496450 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚22781251 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1005956801402 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 1009672423424 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚69185540332806 _runtime‚ 901.020385146 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚71018415323648 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚71018925754368 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 1009672423424 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚22766060 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚76473934886400 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 1010164851200 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15193 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚5831573 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0®Û=0*ŪT JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚156 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚68179092620288 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9954819072 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚69185032382330 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚16039405 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚17496450 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚22630 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚16039249 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚71018415323648 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15191 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚1208 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚5831727 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510430720 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚22903 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 1062021939712 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚70008742900224 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1010164851200 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚70008760903168 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚17496450 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚22781251 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1005956801402 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 1009672423424 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚69185540332806 _runtime‚ 901.020385146 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚71018415323648 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚71018925754368 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 1009672423424 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚22766060 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚76473934886400 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 1010164851200 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15193 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚5831573 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15191 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚70008760903168 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚10207678 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 1006447712518 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490911116 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚40262510 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 135492096 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096`döu²tÆbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP9R ŸrÇbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP‘%"cÈb^ ;DeviceStatsMonitor.on_validation_batch_end/active.all.freed"trainer/global_step2:JPmrŒxpÉbk HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPLKA¡kÊbf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPÿ‚ÑñmËbh EDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current"trainer/global_step2:JP݃2¬uÌbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPÇÞ–pÍbk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JPÐ,ztÎbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPI™’ÚkÏbf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JP˜<¿êmÐbh EDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak"trainer/global_step2:JP¦rFgÑbb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JPcÁ¦ùnÒbi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JPÊÏÍ7oÓbj GDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated"trainer/global_step2:JP ÚêLbÔb] :DeviceStatsMonitor.on_validation_batch_end/active.all.peak"trainer/global_step2:JP[v°rpÕbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP‡½àrÖbm JDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP{ðB.m×bh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JP®£å×rØbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP÷pjÙbe BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JPM¹AynÚbi FDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current"trainer/global_step2:JP ï¤rÛbm JDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPïù¼vÜbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPf!UoÝbj GDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPk*­kÞbf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JPÐÚ`uested_bytes.large_pool.freed"trainer/global_step2:JP6.ôœfëba >DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JPæ &¬pìbk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JP°liÖjíbe BDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak"trainer/global_step2:JP òQemîbh EDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated"trainer/global_step2:JP #àkiïbd ADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed"trainer/global_step2:JPß23Úrðbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPÄ Y4rñbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPáyt-jòbe BDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed"trainer/global_step2:JPÞ›înóbi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JPÂq™ðqôbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPõÄû;qõbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP…Š¢Ìjöbe BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP÷ê¬Ùd÷b_ DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPF£ŠÜk†bf CDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed"trainer/global_step2:JP~$CÊp‡bk HDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPŸ6¤=zˆbu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP¶5–²a‰b\ 9DeviceStatsMonitor.on_validation_batch_end/max_split_size"trainer/global_step2:JPÌAÒÑoŠbj GDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current"trainer/global_step2:JPDZÕ×i‹bd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JPyÄölŒbg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JPß1{nbi FDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated"trainer/global_step2:JPÄŒs„jŽbe BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JP¹\+egbb ?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed"trainer/global_step2:JPT; Fnbi FDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed"trainer/global_step2:JPeipl‘bg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JP¦Q.w’br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPž§,w“br ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP‘Ù_k”bf CDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak"trainer/global_step2:JPZí³t•bo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPÈ_À}k–bf CDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated"trainer/global_step2:JP8GÓt—bo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPíh±¼c˜b^ ;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc"trainer/global_step2:JP×U‚Óq™bl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JPQÖ(¹lšbg DDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current"trainer/global_step2:JPÏ6¯~r›bm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP¾óAîmœbh EDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed"trainer/global_step2:JPe´Q9mbh EDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current"trainer/global_step2:JP–‰o[wžbr ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP"S*ÿiŸbd ADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak"trainer/global_step2:JPM¶Àl bg DDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed"trainer/global_step2:JPfÓà{m¡bh EDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current"trainer/global_step2:JPµ•Z?j¢be BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JP,æ¡°l£bg DDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current"trainer/global_step2:JP¥—.n¤bi FDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed"trainer/global_step2:JPs·Íèt¥bo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPs¢Ñ¢b¦b] :DeviceStatsMonitor.on_validation_batch_end/num_device_free"trainer/global_step2:JP`ç/e§b` =DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections"trainer/global_step2:JP*Öläu¨bp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP¨†PÜv©bq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPñŸ”jªbe BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JPÈÏmq«bl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPYgsús¬bn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPûÒ®3x­bs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPCЬDo®bj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPìÜ¥w¯br ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP¹W1™h°bc @DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak"trainer/global_step2:JP©Svl±bg DDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP¾M»ÿk²bf CDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPT½Yo³bj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JPrÒdCr´bm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JP;èVçmµbh EDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current"trainer/global_step2:JPuÙŸÌs¶bn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP©‚©d·b_ DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚71056120909824 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚71055610479104 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492427776 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚17507256 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚22788553 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15193 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚17507258 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚22773362 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚40280618 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚17507256 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15191 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 1010529977856 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643691477 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 1010529977856 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚40295811 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚155 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚10214180 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 _runtime‚ 901.953508592 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚700455909319684¼ÿš9)óR bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚70045590931968 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚69220689751226 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚22773362 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 trainer/global_step‚1208 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚5836534 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 1010037550080 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚76511810619904 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863048704 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚68214386379776 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9954819072 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚71055610479104 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9954819072 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚157 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1006320410810 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚71056120909824 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚71055610479104 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492427776 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚17507256 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚22788553 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15193 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚17507258 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚22773362 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚40280618 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚17507256 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15191 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 1010529977856 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643691477 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 1010529977856 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚40295811 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚155 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚10214180 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 _runtime‚ 901.953508592aÀ¬ªJÄbE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÍײHÅbC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP‡bJÆbE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPþU}?JÇbE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPÈ¢zNÈbI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPˆL6JÉbE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP¥q­tJÊbE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPX¿MJËbE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPg±F˜JÌbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPaôŽNÍbI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP*FéJÎbE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPz¤»¶HÏbC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP#• ÏJÐbE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPò#7NÑbI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPÏÝ}ïNÒbI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÀÏo;Ób6 jet/accuracy_2_of_4"trainer/global_step2:JP9ÙäNÔbI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPT£‹cHÕbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP“yñÒHÖbC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPü3H×bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPè§JØbE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPËî®NÙbI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP<€ JÚbE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPjqÐÖJÛbE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPY[§JÜbE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP‚oR HÝbC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP,9­³HÞbC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPM2Z£JßbE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP@‘¾JàbE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPSÔÄYNábI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÀ¥%NâbI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPXÄÿJãbE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPo»—ÉNäbI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP„êç•NåbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPmêUôNæbI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP@—Rl;çb6 jet/accuracy_3_of_5"trainer/global_step2:JPJÚÒJèbE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPk¾NébI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP‡PðsNêbI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP³¬Ò\@ëb; particle/accuracy_2_of_5"trainer/global_step2:JPâKª}JìbE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPæc4÷;íb6 jet/accuracy_5_of_7"trainer/global_step2:JPáª5„JîbE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP*ý“8NïbI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPy1ë×JðbE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPo†’JñbE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP~œôc;òb6 jet/accuracy_6_of_6"trainer/global_step2:JPYV²JóbE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP&®HôbC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP>@$NõbI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPªŒ$—HöbC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÅÆÍPH÷bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP–LãJøbE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPŽ¼XNùbI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPcþ†ÈJúbE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP ø²JûbE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPüâJübE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPìg-¥JýbE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŒ™ |NþbI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP/Ž6kHÿbC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP±Ÿù J€bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPyk HbC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP‡šá£H‚bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¶KAgNƒbI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP‘ÏßJ„bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP}íJ…bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP!íäÇH†bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPOÊîŒJ‡bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP–g5¦HˆbC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPÛ¯m'N‰bI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPØf½NŠbI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP’ {N‹bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPä΃¨NŒbI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPBP}NbI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÀ'+^HŽbC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP¯m™HbC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPœáT\NbI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPeï¨Ö@‘b; particle/accuracy_1_of_4"trainer/global_step2:JP \²G’bB validation_average_jet_accuracy"trainer/global_step2:JPä¦H“bC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPU ÎYJ”bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP ‚2XJ•bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPvÀÑ J–bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP'úÚH—bC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP¨n=øH˜bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPg‘ YJ™bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPsøE]JšbE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP1e®N›bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP{x–;œb6 jet/accuracy_1_of_1"trainer/global_step2:JP˜%<b7 particle/sensitivity"trainer/global_step2:JPr,]NžbI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP®=ŒHŸbC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPðÊRH bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPŸ€»ÀH¡bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPÒ2ö;¢b6 jet/accuracy_7_of_7"trainer/global_step2:JPažSN£bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPªï™|J¤bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP—üf N¥bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPLA@NäbI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP(3Ìê;åb6 jet/accuracy_1_of_5"trainer/global_step2:JP‹ƒ?JæbE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPIVA«JçbE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPh‚|ÎJèbE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPpÅ›½HébC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP‘Óá JêbE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP7Oú‡JëbE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPpö&JìbE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPjIòJíbE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPbÝ}ß@îb; particle/accuracy_3_of_5"trainer/global_step2:JP¨g”M@ïb; particle/accuracy_6_of_6"trainer/global_step2:JPb "£JðbE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPþ˜6HñbC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPê sNòbI &Purity/0FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPÜ'”NóbI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPãr"ÔHôbC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPŒ8ËHõbC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP©°_wHöbC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPëà‰ J÷bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPØv×kHøbC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP›TÏ×JùbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPô/‡€NúbI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÀÁ}ÇJûbE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP dΕNübI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPoP– NýbI &Purity/1FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP¶ãPNþbI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPâJÿbE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP!ãõN€ bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPjÜ ·N bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP:-Û5@‚ b; particle/accuracy_4_of_4"trainer/global_step2:JPök ×Jƒ bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP,hŒH„ bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPfKH… bC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPª-"H† bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP̈CH‡ bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPp—g·Hˆ bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPVJ‰ bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPïE ãNŠ bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP1„N‹ bI &Purity/1FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP£‡7HŒ bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÌÍøðH bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP{=#óJŽ bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP+é iN bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPT.ÔN bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÁ£ÕN‘ bI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPEªLN’ bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP ;°J“ bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÐL¬ˆ;” b6 jet/accuracy_4_of_5"trainer/global_step2:JP§nïJ• bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPõQ™J« bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPO9máN¬ bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPë\.N­ bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP`büNÖ bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPéâIôN× bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPxfJÈJØ bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP1X„ÆJÙ bE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPž}¦JÚ bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP‰õÜ[JÛ bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP–àÒJÜ bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPÏà+HÝ bC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPü‡•NÞ bI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÝì:á;ß b6 jet/accuracy_1_of_4"trainer/global_step2:JPNã›vHà bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPI=Ÿò;á b6 jet/accuracy_3_of_6"trainer/global_step2:JPès³:Hâ bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP$ˆyeHã bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPø—Hä bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¿ŒsœJå bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPó®éHæ bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP¨V©Jç bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPyîÀ*Jè bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP¡œ¢Jé bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPö©†%Jê bE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPãØ?žNë bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP?™/Jì bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÌÇüNí bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPèK–õHî bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP¨åZNï bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP*P-Jð bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP[¦§öJñ bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPOb¡Hò bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP°Œz‡Nó bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP#ûûDHô bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÕ=­ÀJõ bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP `°´Jö bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP°K>!J÷ bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPAVDNø bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPjðNù bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP†ñ.<ú b7 particle/specificity"trainer/global_step2:JPˆÉº@û b; particle/accuracy_5_of_6"trainer/global_step2:JP’€X2Hü bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP0_´;ý b6 loss/val_total_loss"trainer/global_step2:JP&UDNþ bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP*˜K—Jÿ bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPœ÷¹ÂH€ bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPÕŒƒêN bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP[‚UJ‚ bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP—›¤€Jƒ bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP‘z:H„ bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPL±±RN… bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP å*6J† bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP#éŠÜN‡ bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÝ%yÄNˆ bI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPó]ËJN‰ bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPNPóJŠ bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP;ŸUÆJ‹ bE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPꡘ±NŒ bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPœîNàN bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPë1µHŽ bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP»oþ%N bI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP|Ž›N bI &Purity/3FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÈ(o;‘ b6 jet/accuracy_1_of_3"trainer/global_step2:JP´šZƒH’ bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP¿AJ“ bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPS­æH” bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPçb%.N• bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP:ö„J– bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP_â9qN— bI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPvöí¸N˜ bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPp±îÇN™ bI &Purity/3FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPzî¢Hš bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPzÍëN› bI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPkBÞ·Hœ bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPìô6‘J bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPĈ"K9ž b4 particle/accuracy"trainer/global_step2:JP\G»HŸ bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPŸm0H  bC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÕÓôúJ¡ bE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP¤E|H¢ bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPXRœH£ bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP×D~ÖJ¤ bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÜÝ;#H¥ bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP„êãJ¦ bE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPc…·J§ bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP ö—yH¨ bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP,´>N© bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPe‘vHª bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP$-÷N« bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP>Þv J¬ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP6eGJ­ bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPþÎ&«N® bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPwöN¯ bI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP;'•N° bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP†²àJ± bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP„Z}N² bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP™ÎÔ N³ bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP8†²Y;´ b6 jet/accuracy_3_of_4"trainer/global_step2:JP/÷XJµ bE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÑC"J¶ bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP /¡PH· bC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP§‡ìÞJ¸ bE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP s H¹ bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÿćJÁ bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPÎü~ÂH bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP&í-Nà bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPĆNÄ bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPwR¯æ;Å b6 jet/accuracy_2_of_5"trainer/global_step2:JP+’}‘@Æ b; particle/accuracy_2_of_7"trainer/global_step2:JPÃãü0JÇ bE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP:ŠÒNÈ bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP­– NÉ bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPX³RÈNÊ bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPrµËGJË bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP‡„Ž]HÌ bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPG×ÔJÍ bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPœ€ÖNÎ bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPú¥ý%NÏ bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP2KÔÁNÐ bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPMæiDHÑ bC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPª¾ç@Ò b; particle/accuracy_4_of_7"trainer/global_step2:JPY^ÊJÓ bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP‘ÿ®JÔ bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP@—ÄHÕ bC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPZÞ°ÆHÖ bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP¨JuÿJ× bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÛê¢HØ bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPÇ_,ìJÙ bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPK^mHÚ bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPr)“üHÛ bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPp°ÚãNÜ bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPi°†cHÝ bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPŽÑ*@Þ b; particle/accuracy_4_of_6"trainer/global_step2:JPG7€ËJß bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPa”gZNà bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP †KNá bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPN#$rHâ bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPö*tSHã bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP*¬š%Hä bC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPëúhæJå bE "Purity/3FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPT·Hæ bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPa ãßJç bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPT`Nè bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPøø!0Jé bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPtïàNê bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP‰1®ÒJë bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP.Àì bI…J’ bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP”ƒE³J“ bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP“BÊbH” bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÅöê\J• bE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPÅSßÌN– bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP¸ù­ J— bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP)š·øN˜ bI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPz”H™ bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP.™]Nš bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP²ÆèJ› bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPš4Hœ bC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP ÛtóH bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPªæ1±Nž bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPf—ÑNŸ bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPû+]íH  bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP”a´*H¡ bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP5ÄH¢ bC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPÌ;B;J£ bE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÍ°xN¤ bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP\‚IH¥ bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP6ö#ÛN¦ bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPù½éN§ bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP¬B‚N¨ bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP*±Rù@© b; particle/accuracy_1_of_2"trainer/global_step2:JPdìhHª bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPPRýJ« bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP2xN¬ bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPTæxh@­ b; particle/accuracy_2_of_4"trainer/global_step2:JPkšJ® bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÀƒm¡N¯ bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPŠ ½H° bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPí;ZH± bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP¶¹ºlN² bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPëu´LH³ bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPT´¾õN´ bI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPŒ3›…Hµ bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPˆÒuÉN¶ bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPôòHN· bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP·ÊaH¸ bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPØ€ˆÞH¹ bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPXq,{Jº bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP{J» bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP»´WH¼ bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP]ÊåN½ bI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPOeB‚N¾ bI &Purity/0FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP–*~6H¿ bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPŸ†!NÀ bI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP°vÆvNÁ bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPŠNãÅH bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPÿ²šHà bC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÃÅ"ÜJÄ bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP±fêÅNÅ bI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPí†t9HÆ bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPâî)ºHÇ bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP¬9‘JÈ bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¼ÿÁHÉ bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPÀEîÕHÊ bC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPÂHNË bI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP€ÉåÍHÌ bC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP,‡m>JÍ bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP*$=×JÎ bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPsŽOuNÏ bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÿÆJÐ bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP™–WNÑ bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP¨ˆÏ­JÒ bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPf$ÖHÓ bC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP$##£NÔ bI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP¾Ö6NÕ bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP°¼GJÖ bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPiõ©‰N× bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPF(NØ bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP_sØJÙ bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPǾo¼NÚ bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPg„Ï£HÛ bC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP¡Ò´ÕHÜ bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPΘ]HÝ bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPŽÖ÷¾HÞ bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP¸ Nß bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP¶jãNà bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP ÊÑÌ;á b6 jet/accuracy_1_of_2"trainer/global_step2:JPÝèÉ”Jâ bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPüO*Jã bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPôÇ2KNä bI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPW »ÏNå bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPªÝßWHæ bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP8òÒNç bI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPWÔh€;è b6 jet/accuracy_4_of_6"trainer/global_step2:JPy }Né bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP zí¼Jê bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¢iêHë bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP¯TJì bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPHÚ bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPÝŠëôJÛ bE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPz˜ØÚHÜ bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPÒ1HÝ bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP2Ý}HÞ bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP¹UÄiHß bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPÚúe”Hà bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPÐMö¶Há bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPTÏuJâ bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPíÙÿRHã bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP¤ ýJä bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP²o–Jå bE "Purity/1FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPpØ}2Jæ bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP«ÎMNç bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP½p1¾Jè bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPi€`Né bI &Purity/4FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPš#4äHê bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPâ-æ(Jë bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPn ¤CJì bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPBãÑ ;í b6 jet/accuracy_2_of_6"trainer/global_step2:JP[À^Jî bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPPðszJï bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP n6ø@ð b; particle/accuracy_3_of_7"trainer/global_step2:JPÀ»úJñ bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPVÒ0Nò bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPAìd‘Nó bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP„·e‹Jô bE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPžá)QJõ bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPɲ[;ö b6 jet/accuracy_6_of_7"trainer/global_step2:JP‚42»N÷ bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP!1 Nø bI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPþm¯ONù bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP™cG½Jú bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPŸ›ž©Hû bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPt£.uNü bI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPb5 Jý bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPo­ŸßJþ bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP™ÿ#Nÿ bI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPgÄeN€ bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP0aN bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP¡XæþN‚ bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPöŒª¶Nƒ bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP®ê3Š@„ b; particle/accuracy_3_of_4"trainer/global_step2:JP¾{Õâ;… b6 jet/accuracy_2_of_7"trainer/global_step2:JPe­¯pJ† bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP–íÕN‡ bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP†/•Jˆ bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPfh0J‰ bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP€»mÍHŠ bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP#Ö7eJ‹ bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP[¿÷@Œ b; particle/accuracy_2_of_2"trainer/global_step2:JPÍh2Ï; b6 jet/accuracy_5_of_6"trainer/global_step2:JP!äHŽ bC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPXïdáH bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP‹Q>H bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP®W*oH‘ bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPP¦ÁÇH’ bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPys0“ + _wandbruntime‚914  _runtime‚914yÕM'*#” “¥ 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3813595175743103 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.17014984786510468 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9837151169776917 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.2652851343154907 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.17499999701976776 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3691149950027466 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.015930218622088432 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1676303744316101 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.05303030088543892 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.24137932062149048 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.37560802698135376 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3046130836009979 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5681818127632141 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.21522988379001617 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.014901050366461277 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03127281740307808 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17040227353572845 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.7884128093719482 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.06666666269302368 +jet/accuracy_3_of_6‚0.07692307978868484 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9861701130867004 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9933676719665527 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.20122887194156647 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011791384778916836 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2258608043193817 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.34210526943206787 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.005228238180279732 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2563066780567169 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.24089105427265167 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9308363795280457 particle/accuracy_5_of_6‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 )loss/val_total_loss‚61.39990908266852 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.20433008670806885 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07143957167863846 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1381993144750595 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7065562605857849 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2292080521583557 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1550017148256302 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2284717708826065 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18304093182086945 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3276785910129547 (particle/accuracy‚0.8274170160293579 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.12191358208656311 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.19179369509220123 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2020667940378189 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9657385945320129 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.22008992731571198 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.19953688979148865 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.35436126589775085 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.03673538193106651 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9884663820266724 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2454860955476761 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 jet/accuracy_5_of_5‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2708333432674408 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.26287204027175903 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 +jet/accuracy_2_of_5‚0.32952505350112915 particle/accuracy_2_of_7‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 particle/accuracy_4_of_7‚1 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0062443045899271965 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.09563098847866058 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.27489495277404785 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.20274315774440765 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3214285671710968 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2480815052986145 >&Purity/2FRt*SRqqt1FBt/event_proportionè;X9po‚0.02114335261285305 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12563642859458923 particle/accuracy_4_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11088530719280243 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.25912901759147644 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.09977413713932037 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03448275849223137 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0505356527864933 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.002310114214196801 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6619948148727417 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14844879508018494 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 particle/accuracy_3_of_3‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1697530895471573 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.33689942955970764 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.07741042226552963 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15353699028491974 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9857364892959595 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4385965168476105 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.017473118379712105 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9853271245956421 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.27000001072883606 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.125 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13848905265331268 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3217490613460541 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.10684995353221893 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07324553281068802 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12864328920841217 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9881685972213745 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.996428370475769 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.033018868416547775 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.3898772895336151 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16148647665977478 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2386224865913391 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.1675925999879837 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1100025400519371 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5378787517547607 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.19135801494121552 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.18819159269332886 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.24278470873832703 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2083333134651184 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9901250600814819 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970021843910217 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1486581712961197 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.5789070725440979 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.15421457588672638 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3445141911506653 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02380952425301075 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.3266654312610626 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.18082787096500397 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.26878976821899414 jet/accuracy_3_of_7‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9669539928436279 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.13312171399593353 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1082395538687706 particle/accuracy_1_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16140830516815186 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.10586906224489212 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15833333134651184 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 jet/accuracy_4_of_7‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.05543654039502144 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 jet/accuracy_1_of_7‚0.5 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.35914042592048645 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1501743495464325 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.3410802185535431 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0045042261481285095 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 % _timestamp‚1.7884292295684667e+09 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.25924643874168396 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14448219537734985 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.07237537950277328 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,jet/accuracy_3_of_3‚0.013180031441152096 *jet/accuracy_1_of_6‚0.8846153616905212 particle/accuracy_3_of_6‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9302600026130676 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 +jet/accuracy_2_of_3‚0.18004438281059265 particle/accuracy_2_of_3‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.2866666615009308 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.37656524777412415 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.3125 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.13993993401527405 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 _step‚98 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11044127494096756 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2288060337305069 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 particle/accuracy_2_of_6‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14323130249977112 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.23245613276958466 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006613756995648146 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.002807604381814599 particle/accuracy_5_of_5‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1875 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8102859258651733 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.201388880610466 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.0555555559694767 particle/accuracy_7_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07092198729515076 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.38041749596595764 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19476114213466644 , Purity/1FRt1SRqqt2FBt/FBt_purity‚0.96875 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.34196150302886963 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.20992445945739746 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2507054805755615 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3816707730293274 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.3461538553237915 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.17549018561840057 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 particle/accuracy_3_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.14848485589027405 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.25072941184043884 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.242248073220253 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_3_of_4‚1 jet/accuracy_2_of_7‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16592153906822205 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2163921743631363 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_2‚1 jet/accuracy_5_of_6‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.011900532990694046 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1944444477558136 trainer/global_step‚1208 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3585570156574249 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3459770083427429 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0019841270986944437 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_2_of_4‚0.2730449140071869 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15253117680549622 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9927248954772949 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9933862686157227 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.130898118019104 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.25757578015327454 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3849206268787384 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.18352963030338287 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08794162422418594 epoch‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.005536084994673729 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2395833283662796 jet/accuracy_5_of_7‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2708333432674408 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3065972328186035 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 jet/accuracy_6_of_6‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9921955466270447 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.1920638382434845 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1825396716594696 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9866029024124146 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9937068223953247 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.11749356985092163 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18423837423324585 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.3228457570075989 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166666269302368 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3235294222831726 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16610939800739288 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.20796319842338562 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.34063249826431274 ,particle/sensitivity‚0.26834410429000854 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2043103277683258 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2261846661567688 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.21396416425704956 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9823785424232483 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2253086417913437 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.0795469731092453 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.21189086139202118 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33110350370407104 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.2442302703857422 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6157324910163879 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.2343738079071045 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06022302061319351 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0912698432803154 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.006235827691853046 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2425890415906906 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.20934908092021942 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9914952516555786 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.37036049365997314 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9968729615211487 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16010329127311707 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.36110833287239075 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24929170310497284 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9903031587600708 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.43620869517326355 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.17794904112815857 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31865671277046204 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.2261846661567688 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.10925926268100739 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3310185372829437 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_5‚0.8452189564704895 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 +"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.1892819106578827 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07143957167863846 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.24441982805728912 particle/accuracy_3_of_5‚1 particle/accuracy_6_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3426115810871124 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_4_of_4‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2465570718050003 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14052079617977142 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3878071904182434 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 _runtime‚ 914.275454638 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2085152119398117 -jet/accuracy_4_of_5‚0.0026881720405071974 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_1_of_7‚1 particle/accuracy_5_of_7‚1 particle/accuracy_6_of_7‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9847233891487122 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9917742609977722 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0667036920785904 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09517953544855118 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2577815651893616 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0017479965463280678 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.06666667014360428 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0&Ì9‚• —¥ @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9921955466270447 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.1920638382434845 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1825396716594696 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9866029024124146 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9937068223953247 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.11749356985092163 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18423837423324585 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.3228457570075989 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166666269302368 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3235294222831726 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16610939800739288 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.20796319842338562 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.34063249826431274 ,particle/sensitivity‚0.26834410429000854 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2043103277683258 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2261846661567688 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.21396416425704956 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9823785424232483 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2253086417913437 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.0795469731092453 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.21189086139202118 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33110350370407104 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.2442302703857422 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6157324910163879 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.2343738079071045 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06022302061319351 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0912698432803154 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.006235827691853046 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2425890415906906 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.20934908092021942 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.99149525165®Ä&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_1_of_7‚1 particle/accuracy_5_of_7‚1 particle/accuracy_6_of_7‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9847233891487122 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9917742609977722 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0667036920785904 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09517953544855118 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2577815651893616 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0017479965463280678 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.06666667014360428 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3813595175743103 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.17014984786510468 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9837151169776917 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.2652851343154907 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.17499999701976776 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3691149950027466 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.015930218622088432 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1676303744316101 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.05303030088543892 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.24137932062149048 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.37560802698135376 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3046130836009979 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5681818127632141 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.21522988379001617 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.014901050366461277 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03127281740307808 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17040227353572845 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.7884128093719482 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.06666666269302368 +jet/accuracy_3_of_6‚0.07692307978868484 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9861701130867004 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9933676719665527 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.20122887194156647 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011791384778916836 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2258608043193817 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.34210526943206787 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.005228238180279732 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2563066780567169 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.24089105427265167 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9308363795280457 particle/accuracy_5_of_6‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 )loss/val_total_loss‚61.39990908266852 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.20433008670806885 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07143957167863846 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1381993144750595 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7065562605857849 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2292080521583557 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1550017148256302 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2284717708826065 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18304093182086945 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3276785910129547 (particle/accuracy‚0.8274170160293579 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.12191358208656311 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.19179369509220123 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2020667940378189 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9657385945320129 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.22008992731571198 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.19953688979148865 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.35436126589775085 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.03673538193106651 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9884663820266724 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2454860955476761 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 jet/accuracy_5_of_5‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2708333432674408 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.26287204027175903 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 +jet/accuracy_2_of_5‚0.32952505350112915 particle/accuracy_2_of_7‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 particle/accuracy_4_of_7‚1 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0062443045899271965 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.09563098847866058 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.27489495277404785 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.20274315774440765 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3214285671710968 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2480815052986145 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12563642859458923 particle/accuracy_4_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11088530719280243 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.25912901759147644 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.09977413713932037 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03448275849223137 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0505356527864933 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.002310114214196801 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6619948148727417 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14844879508018494 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 particle/accuracy_3_of_3‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1697530895471573 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.33689942955970764 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.07741042226552963 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15353699028491974 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9857364892959595 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4385965168476105 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.017473118379712105 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9853271245956421 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.27000001072883606 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.125 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13848905265331268 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3217490613460541 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.10684995353221893 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07324553281068802 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12864328920841217 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9881685972213745 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.996428370475769 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.033018868416547775 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.3898772895336151 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16148647665977478 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2386224865913391 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.1675925999879837 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1100025400519371 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5378787517547607 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.19135801494121552 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.18819159269332886 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.24278470873832703 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2083333134651184 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9901250600814819 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970021843910217 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1486581712961197 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.5789070725440979 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.15421457588672638 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3445141911506653 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02380952425301075 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.3266654312610626 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.18082787096500397 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.26878976821899414 jet/accuracy_3_of_7‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9669539928436279 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.13312171399593353 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1082395538687706 particle/accuracy_1_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16140830516815186 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.10586906224489212 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15833333134651184 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 jet/accuracy_4_of_7‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.05543654039502144 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 jet/accuracy_1_of_7‚0.5 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.35914042592048645 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1501743495464325 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.3410802185535431 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0045042261481285095 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 % _timestamp‚1.7884292295684667e+09 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.25924643874168396 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14448219537734985 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.07237537950277328 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,jet/accuracy_3_of_3‚0.013180031441152096 *jet/accuracy_1_of_6‚0.8846153616905212 particle/accuracy_3_of_6‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9302600026130676 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 +jet/accuracy_2_of_3‚0.18004438281059265 particle/accuracy_2_of_3‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.2866666615009308 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.37656524777412415 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.3125 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.13993993401527405 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 _step‚98 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11044127494096756 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2288060337305069 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 particle/accuracy_2_of_6‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14323130249977112 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.23245613276958466 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006613756995648146 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.002807604381814599 particle/accuracy_5_of_5‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1875 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8102859258651733 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.201388880610466 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.0555555559694767 particle/accuracy_7_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07092198729515076 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.38041749596595764 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19476114213466644 , Purity/1FRt1SRqqt2FBt/FBt_purity‚0.96875 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.34196150302886963 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.20992445945739746 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2507054805755615 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3816707730293274 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.3461538553237915 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.17549018561840057 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 particle/accuracy_3_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.14848485589027405 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.25072941184043884 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.242248073220253 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_3_of_4‚1 jet/accuracy_2_of_7‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16592153906822205 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2163921743631363 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_2‚1 jet/accuracy_5_of_6‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.011900532990694046 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1944444477558136 trainer/global_step‚1208 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3585570156574249 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3459770083427429 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0019841270986944437 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_2_of_4‚0.2730449140071869 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15253117680549622 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9927248954772949 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9933862686157227 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.130898118019104 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.25757578015327454 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3849206268787384 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.18352963030338287 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08794162422418594 epoch‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.005536084994673729 ,&Purity/1FRt3SRçPðD#qqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2395833283662796 jet/accuracy_5_of_7‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2708333432674408 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3065972328186035 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 jet/accuracy_6_of_6‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0b+–@’‘– ‚(:† ®‡åÔÊúø gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì™N#— ‚(:˜ ®‡åÔ›áÏ network.recv‚397053% disk./.usageGB‚1098.7950019836426 disk.drbd1777.in‚0 disk.dm-0.out‚3837.673828125 disk.nvme0n1.in‚0 disk.dm-0.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚3837.673828125 disk.rbd1.out‚ 0.61328125% disk.nvme0c0n1.out‚3837.673828125 disk.sda2.in‚0( proc.memory.availableMB‚ 285.19140625" proc.memory.rssMB‚ 2868.3671875 disk.sda3.in‚0 disk.rbd4.out‚0 disk.sda.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.507123947143555 proc.cpu.threads‚17) disk./.usagePercent‚73.74105453353707 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 38.5859375 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 cpu‚25.359641028574323# memory_percent‚98.2593297958374 disk.sda3.out‚ 38.5859375 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda.out‚ 38.5859375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚53.33447265625 disk.rbd3.in‚0 disk.rbd1.in‚0 network.sent‚ 1.508576e+06 disk.rbd3.out‚0Id—ž˜ ‚(:“ ½‡åÔó§þ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚102.499' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.499799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*zs!™ ‚(:– ½‡åÔ–Õ³ disk.rbd4.in‚0# disk.nvme0n1.out‚4882.880859375 disk.nvme0c0n1.in‚0 network.recv‚404213( disk./.usagePercent‚73.8030485500061 disk.sda3.in‚0 disk.rbd4.out‚0 disk.sda.out‚ 39.38671875 disk.rbd1.out‚ 0.61328125 disk.sda2.in‚0' proc.memory.availableMB‚ 960.0546875% disk./.usageGB‚1099.7187576293945 disk.rbd2.out‚53.47119140625 disk.dm-1.in‚0 disk.nvme0n1.in‚0 disk.dm-0.out‚4882.880859375! proc.memory.rssMB‚ 2862.515625 network.sent‚ 1.65393e+06 disk.rbd2.in‚0 disk.sda3.out‚ 39.38671875 disk.drbd1777.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 39.38671875 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0) proc.memory.percent‚17.47140884399414 cpu‚25.35531760450198 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚4882.880859375 disk.rbd0.out‚0$ memory_percent‚94.14029121398926Dèæþžš ‚(:“ ̇åÔ“䌎 gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚103.862' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.772399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NŠóÀ%› ‚(:š ̇åÔá½Ñ disk.sda3.in‚0 disk.sda.in‚0& disk.nvme0c0n1.out‚4982.1435546875 cpu‚25.350331001138628 disk.rbd3.in‚0! disk.dm-0.out‚4982.1435546875 disk.sda.out‚ 39.66015625 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0) proc.memory.percent‚17.47140884399414 network.sent‚ 1.659325e+06% disk./.usageGB‚1099.7188148498535 disk.sda3.out‚ 39.66015625 disk.rbd4.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 39.66015625 disk.rbd1.out‚ 0.61328125 disk.rbd4.out‚0 network.recv‚408120( disk./.usagePercent‚73.8030523901191 disk.rbd2.out‚53.47119140625 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚4982.1435546875 disk.sda2.out‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 960.203125! proc.memory.rssMB‚ 2862.515625 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚0$ memory_percent‚94.13938522338867à†Š'0œ + _wandbruntime‚947  _runtime‚947éƒK9Æ À trainer/global_step‚1249 $ _timestamp‚1.788429262321264e+09 _runtime‚ 947.026297351 _step‚99 ' lr-AdamW/pg1‚0.00022661525021142121 ' lr-AdamW/pg2‚0.00022661525021142121’Íè¯Êž Ä trainer/global_step‚1249 $ _timestamp‚1.788429262321264e+09 _runtime‚ 947.026297351 _step‚99 ' lr-AdamW/pg1‚0.00022661525021142121 ' lr-AdamW/pg2‚0.00022661525021142121cO›œ0Ÿ + _wandbruntime‚947  _runtime‚947¼”ãÆ'  ÀO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1047414203392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1047414203392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72778502860288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚428 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23627720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10625454 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10625452 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6076020 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41836096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70869353436160 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚70869370475520 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23607684 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71913580806911 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73825899060736 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚73825899060736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1048070564864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18208374 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18208376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚ 947.027403686 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23607684 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1043555893643 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79470248602112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884292623222427e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1044210331391 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73826573425152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23627720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157333504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41816058 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1048070564864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18208376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18208374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚100 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72778484857344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72778484857344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6075594 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80571635166208 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚1249s(Ê'¡ ÄO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41816058 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1048070564864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18208376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18208374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚100 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72778484857344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72778484857344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6075594 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80571635166208 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚1249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1047414203392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1047414203392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72778502860288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚428 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23627720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10625454 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10625452 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6076020 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41836096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70869353436160 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚70869370475520 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23607684 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71913580806911 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73825899060736 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚73825899060736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1048070564864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18208374 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18208376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚ 947.027403686 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23607684 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1043555893643 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79470248602112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884292623222427e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1044210331391 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73826573425152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23627720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157333504dj©0¢ + _wandbruntime‚947  _runtime‚947fDk¾Ž£ ÊM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10633622 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70924274606080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73883745905152 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41848528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1102433002496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16713703 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80634332342272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41868592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72834837896192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1048233631232 % _timestamp‚1.7884292629884713e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72834855899136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6080083 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41848528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72834837896192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41868592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23626138 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72834855899136 trainer/global_step‚1249 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1044372328607 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚73883071527424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71969301372643 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18222390 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79531899339776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚101 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73883745905152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23626138 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23646200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1048233631232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚ 947.693972534 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1048890006016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18222392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 NAUÛ-B DeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18222392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1048890006016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1102271488000 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70924257566720 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71968629895327 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18222390 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79531875399680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23646200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10633620 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161514496 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73883745905152 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41848528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1102433002496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16713703 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80634332342272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41868592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72834837896192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1048233631232 % _timestamp‚1.7884292629884713e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72834855899136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6080083 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41848528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72834837896192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41868592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23626138 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72834855899136 trainer/global_step‚1249 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1044372328607 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚73883071527424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71969301372643 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18222390 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79531899339776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚101 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73883745905152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23626138 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23646200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1048233631232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚ 947.693972534 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1048890006016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18222392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18222392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1048890006016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1102271488000 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70924257566720 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71968629895327 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18222390 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79531875399680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513e!·Ð0¥ + _wandbruntime‚947  _runtime‚947ÐBIeR¦ Ì /loss/FRt4/detection_loss‚2.2839175709362687 1loss/SRqqt2/detection_loss‚2.0168722195828215 _runtime‚ 947.694951008 /loss/FRt4/assignment_loss‚4.352187867822318 1loss/SRqqt2/assignment_loss‚2.441546857991117 0loss/FBt3/assignment_loss‚0.6053483485681378 .loss/FRt2/detection_loss‚2.241190888263561 trainer/global_step‚1249 /loss/FRt2/assignment_loss‚4.751570317480299 /loss/FBt4/detection_loss‚3.2802317615817573 2loss/SRqqt3/assignment_loss‚3.0639418471943247 /loss/FBt1/assignment_loss‚0.290518869118047 1loss/SRqqt3/detection_loss‚3.0575468865307895 0loss/SRqqt4/detection_loss‚2.191508733948996 /loss/FBt1/detection_loss‚0.8286680533760546 /loss/FBt3/detection_loss‚3.8040586709976196 /loss/FRt1/assignment_loss‚2.647199898373847 %loss/total_loss‚59.97823408106195 2loss/SRqqt1/assignment_loss‚2.1032423231750728 1loss/SRqqt1/detection_loss‚1.2718766667880117 epoch‚1 % _timestamp‚1.7884292629902508e+09 /loss/FRt3/assignment_loss‚5.458416287715618 1loss/SRqqt4/assignment_loss‚2.284757525421852 0loss/FBt2/assignment_loss‚0.2178622669827746 .loss/FRt1/detection_loss‚2.224605549784268 .loss/FRt3/detection_loss‚3.146695463703229 _step‚102 0loss/FBt4/assignment_loss‚0.5995547884583229 .loss/FBt2/detection_loss‚4.814914417266846Q}OXV§ Ð 1loss/SRqqt2/assignment_loss‚2.441546857991117 0loss/FBt3/assignment_loss‚0.6053483485681378 .loss/FRt2/detection_loss‚2.241190888263561 trainer/global_step‚1249 /loss/FRt2/assignment_loss‚4.751570317480299 /loss/FBt4/detection_loss‚3.2802317615817573 2loss/SRqqt3/assignment_loss‚3.0639418471943247 /loss/FBt1/assignment_loss‚0.290518869118047 1loss/SRqqt3/detection_loss‚3.0575468865307895 0loss/SRqqt4/detection_loss‚2.191508733948996 /loss/FBt1/detection_loss‚0.8286680533760546 /loss/FBt3/detection_loss‚3.8040586709976196 /loss/FRt1/assignment_loss‚2.647199898373847 %loss/total_loss‚59.97823408106195 2loss/SRqqt1/assignment_loss‚2.1032423231750728 1loss/SRqqt1/detection_loss‚1.2718766667880117 epoch‚1 % _timestamp‚1.7884292629902508e+09 /loss/FRt3/assignment_loss‚5.458416287715618 1loss/SRqqt4/assignment_loss‚2.284757525421852 0loss/FBt2/assignment_loss‚0.2178622669827746 .loss/FRt1/detection_loss‚2.224605549784268 .loss/FRt3/detection_loss‚3.146695463703229 _step‚102 0loss/FBt4/assignment_loss‚0.5995547884583229 .loss/FBt2/detection_loss‚4.814914417266846 /loss/FRt4/detection_loss‚2.2839175709362687 1loss/SRqqt2/detection_loss‚2.0168722195828215 _runtime‚ 947.694951008 /loss/FRt4/assignment_loss‚4.352187867822318f @s]ž¨ ‚(:“ Û‡åÔëÄŸ gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚149.292' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.858400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WE7…&© ‚(:› Û‡åÔ袮‘% disk./.usageGB‚1099.7187576293945 disk.rbd3.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚55.18603515625 disk.rbd4.in‚0 disk.sda.in‚0) proc.memory.percent‚17.47140884399414 disk.rbd2.in‚0 disk.sda3.out‚ 40.14453125 disk.rbd3.out‚0 disk.rbd4.out‚0! disk.dm-0.out‚5081.2333984375! proc.memory.rssMB‚ 2862.515625 disk.rbd1.out‚ 0.61328125 disk.sda3.in‚0 disk.sda.out‚ 40.14453125 disk.dm-0.in‚0 network.sent‚ 1.665659e+06 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚5081.2333984375 disk.rbd0.in‚0 disk.drbd1777.out‚0 network.recv‚412534( disk./.usagePercent‚73.8030485500061 disk.rbd1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 40.14453125$ disk.nvme0n1.out‚5081.2333984375$ memory_percent‚94.14923191070557 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 958.58984375 cpu‚25.34572269702641 proc.cpu.threads‚17j¢÷žª ‚(:“ ê‡åÔº¡Ü gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.606' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.121199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿  ¼&« ‚(:› ê‡åÔĺ» disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚5081.77294921875 disk.rbd1.out‚ 0.61328125 network.sent‚ 1.773021e+06 network.recv‚418562( disk./.usagePercent‚73.8030523901191 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚0$ memory_percent‚94.14939880371094' disk.nvme0c0n1.out‚5081.77294921875 disk.sda2.in‚0 disk.rbd2.out‚55.18603515625 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 40.55859375 disk.sda1.out‚0! proc.memory.rssMB‚ 2862.515625) proc.memory.percent‚17.47140884399414 disk.rbd3.out‚0$ proc.memory.availableMB‚958.5625 cpu‚25.341339015724852 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚0" disk.dm-0.out‚5081.77294921875 disk.sda.out‚ 40.55859375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 40.55859375% disk./.usageGB‚1099.7188148498535 v‡u0¬ + _wandbruntime‚982  _runtime‚982) øÅ­ ¿ $ _timestamp‚1.788429297655082e+09 _runtime‚ 982.359905016 _step‚103 & lr-AdamW/pg1‚0.0002265845536422036 & lr-AdamW/pg2‚0.0002265845536422036 trainer/global_step‚1299F|1¤É® à & lr-AdamW/pg1‚0.0002265845536422036 & lr-AdamW/pg2‚0.0002265845536422036 trainer/global_step‚1299 $ _timestamp‚1.788429297655082e+09 _runtime‚ 982.359905016 _step‚103g¢k Î0¯ + _wandbruntime‚982  _runtime‚982ì0LÆ'° ÀO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6302438 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82552761545216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚75596154802688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75596136799744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚ 982.360923555 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43462948 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18909174 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74698951197211 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18909174 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6302016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76684536808960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1085045670799 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24553772 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11033854 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884292976559367e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11033852 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1089056370688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24533736 _step‚104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73614577003520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24553772 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17336292 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1144454535168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚75596154802688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17335868 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1088400009216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1144288813056 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1089056370688 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚1299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚83697240020480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1088400009216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.smaoé@]ll_pool.current‚ 165722112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1144288813056 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1089056370688 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚1299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚83697240020480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18909174 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74698951197211 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18909174 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6302016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76684536808960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1085045670799 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24553772 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11033854 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884292976559367e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11033852 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1089056370688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24533736 _step‚104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73614577003520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24553772hc ¥0² + _wandbruntime‚983  _runtime‚983=7Ð&³ ÊM R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76741709275648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18923190 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚75652489838592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚75652489838592 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24572252 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1089219437056 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43475380 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18923190 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75652507841536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82614436222976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74754671762735 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _runtime‚ 983.031954516 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43475380 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43495444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11042022 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74755343240051 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73669481134080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24552190 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17348513 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1085207668015 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1145318011392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚105 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18923192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83759754234368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183357440 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83759570876928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159417344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚426 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76742383653376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75652507841536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1299 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43495444 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18923192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1299 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43495444 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18923192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24572252 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1089219437056 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43475380 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18923190 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75652507841536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82614436222976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74754671762735 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _runtime‚ 983.031954516 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43475380 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43495444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11042022 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74755343240051 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73669481134080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24552190 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17348513 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1085207668015 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1145318011392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚105 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18923192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83759754234368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183357440 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83759570876928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159417344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚426 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0i3`0µ + _wandbruntime‚983  _runtime‚983f×~XT¶ Î /loss/FRt2/assignment_loss‚4.398280160294639 /loss/FRt4/detection_loss‚2.4290663615653387 0loss/SRqqt3/detection_loss‚3.020474038521449 /loss/FBt4/detection_loss‚3.9658400590221086 1loss/SRqqt3/assignment_loss‚2.170627695322037 .loss/FRt3/detection_loss‚3.002787980341142 trainer/global_step‚1299 2loss/SRqqt4/assignment_loss‚2.0214272274502685 1loss/SRqqt2/detection_loss‚2.1331031203269957 $loss/total_loss‚63.6612364349817 epoch‚1 % _timestamp‚1.7884292983287103e+09 0loss/FRt1/assignment_loss‚2.7287440024040364 /loss/FRt3/assignment_loss‚5.240796489100302 /loss/FRt2/detection_loss‚2.8945173770189285 .loss/FBt2/detection_loss‚5.367622931798299 2loss/SRqqt2/assignment_loss‚2.2366459023952485 0loss/FBt1/assignment_loss‚0.2839759587906286 0loss/FBt2/assignment_loss‚0.7307006173456708 /loss/FRt1/detection_loss‚2.2442420838331736 /loss/FBt1/detection_loss‚1.2202414694580095 /loss/FRt4/assignment_loss‚4.647679030895233 2loss/SRqqt1/assignment_loss‚1.8876872963765088 0loss/FBt3/assignment_loss‚0.5024312417954206 1loss/SRqqt4/detection_loss‚2.0542906852705136 .loss/FBt3/detection_loss‚6.577360183000565 _runtime‚ 983.033323876 _step‚106 0loss/FBt4/assignment_loss‚0.5228664968162763 0loss/SRqqt1/detection_loss‚1.379828025838908KÓ‘éX· Ò 2loss/SRqqt2/assignment_loss‚2.2366459023952485 0loss/FBt1/assignment_loss‚0.2839759587906286 0loss/FBt2/assignment_loss‚0.7307006173456708 /loss/FRt1/detection_loss‚2.2442420838331736 /loss/FBt1/detection_loss‚1.2202414694580095 /loss/FRt4/assignment_loss‚4.647679030895233 2loss/SRqqt1/assignment_loss‚1.8876872963765088 0loss/FBt3/assignment_loss‚0.5024312417954206 1loss/SRqqt4/detection_loss‚2.0542906852705136 .loss/FBt3/detection_loss‚6.577360183000565 _runtime‚ 983.033323876 _step‚106 0loss/FBt4/assignment_loss‚0.5228664968162763 0loss/SRqqt1/detection_loss‚1.379828025838908 /loss/FRt2/assignment_loss‚4.398280160294639 /loss/FRt4/detection_loss‚2.4290663615653387 0loss/SRqqt3/detection_loss‚3.020474038521449 /loss/FBt4/detection_loss‚3.9658400590221086 1loss/SRqqt3/assignment_loss‚2.170627695322037 .loss/FRt3/detection_loss‚3.002787980341142 trainer/global_step‚1299 2loss/SRqqt4/assignment_loss‚2.0214272274502685 1loss/SRqqt2/detection_loss‚2.1331031203269957 $loss/total_loss‚63.6612364349817 epoch‚1 % _timestamp‚1.7884292983287103e+09 0loss/FRt1/assignment_loss‚2.7287440024040364 /loss/FRt3/assignment_loss‚5.240796489100302 /loss/FRt2/detection_loss‚2.8945173770189285 .loss/FBt2/detection_loss‚5.367622931798299jTj\¸ž¸ ‚(:“ ù‡åÔóσŽ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.802999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó€š"¹ ‚(:— ù‡åÔÉÞØ disk.rbd1.out‚ 0.61328125 disk.nvme0c0n1.in‚0 network.sent‚ 1.779351e+06 network.recv‚423362% disk./.usageGB‚1099.7189979553223 disk.sda3.out‚ 41.390625 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda.out‚ 41.390625 disk.dm-1.in‚0 disk.nvme0n1.in‚0 disk.rbd0.in‚0$ memory_percent‚94.15578842163086 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 55.3544921875! proc.memory.rssMB‚ 2862.515625 disk.rbd3.out‚0 disk.dm-0.in‚0" disk.dm-0.out‚5180.89404296875% disk.nvme0n1.out‚5180.89404296875 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 41.390625 disk.rbd0.out‚0) proc.memory.percent‚17.47140884399414) disk./.usagePercent‚73.80306467848075 proc.cpu.threads‚17 disk.rbd4.out‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 957.515625 disk.sda1.out‚0' disk.nvme0c0n1.out‚5180.89404296875 cpu‚25.337025630588045ÑDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79543174557184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11442252 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85635322368512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179176448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78413806745088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79543174557184 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17971560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19609974 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86822591273984 _step‚108 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86822412097536 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77484993064619 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1129385815040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1125881010207 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1187268905472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 1018.21625874 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1130042176512 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78413788742144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19609976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1129385815040 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1187113669120 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1129385815040 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1187113669120 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79543174557184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11442252 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85635322368512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179176448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78413806745088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79543174557184 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17971560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19609974 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86822591273984 _step‚108 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86822412097536 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77484993064619 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1129385815040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1125881010207 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1187268905472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0l'ÊC2 - _wandbruntime‚1018  _runtime‚1018ö7–ãÑ&à ËM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45122296 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17984693 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6534271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85696973106176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1130205242880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1126043007423 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25498304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25498304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45122296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19623990 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11450420 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25478242 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77540713630143 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17984248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚109 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78470159783936 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79600347023872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76414687662080 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45102232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19623992 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19623990 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11450422 _runtime‚1018.897056198 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1130205242880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78470141780992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1130861617664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6533828 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚1349 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77541385107459 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25478242 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1187974833664 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79601021401600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86884923999744 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76414670622720 % _timestamp‚1.7884293341917925e+09CÈü_Ä ÏM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78470141780992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1130861617664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6533828 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚1349 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77541385107459 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25478242 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1187974833664 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79601021401600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86884923999744 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76414670622720 % _timestamp‚1.7884293341917925e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45122296 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17984693 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6534271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85696973106176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1130205242880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1126043007423 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25498304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25498304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45122296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19623990 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11450420 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25478242 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77540713630143 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17984248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚109 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78470159783936 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79600347023872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76414687662080 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45102232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19623992 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19623990 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11450422 _runtime‚1018.897056198 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1130205242880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656mÜß Y2Å - _wandbruntime‚1018  _runtime‚1018æò†ˆTÆ Î /loss/FRt2/assignment_loss‚4.789680123329163 0loss/FBt4/assignment_loss‚0.6338688023388386 1loss/SRqqt1/detection_loss‚1.2151799519824518 1loss/SRqqt2/detection_loss‚2.5837761000279458 2loss/SRqqt1/assignment_loss‚1.9178226117963915 .loss/FBt2/detection_loss‚4.545104891061783 epoch‚1 0loss/FRt1/assignment_loss‚2.9451241029633417 2loss/SRqqt2/assignment_loss‚2.3502155523146353 1loss/FBt1/assignment_loss‚0.33672770426040916 _runtime‚1018.898746824 .loss/FRt3/assignment_loss‚5.42288925008076 2loss/SRqqt4/assignment_loss‚2.2273596860468388 /loss/FRt1/detection_loss‚1.9578814927547696 .loss/FRt2/detection_loss‚2.658751839815184 /loss/FRt4/detection_loss‚2.2744767974162925 1loss/SRqqt3/detection_loss‚2.7132514227520335 0loss/SRqqt4/detection_loss‚1.757147935398838 &loss/total_loss‚61.281267651865065 0loss/FBt2/assignment_loss‚0.5860430779866874 /loss/FBt3/assignment_loss‚0.725798743243594 .loss/FRt3/detection_loss‚2.665621881077929 trainer/global_step‚1349 _step‚110 /loss/FRt4/assignment_loss‚4.678150144116632 /loss/FBt1/detection_loss‚0.8871377822402114 .loss/FBt3/detection_loss‚4.462595891207457 .loss/FBt4/detection_loss‚4.379087708890438 # _timestamp‚1.78842933419408e+09 2loss/SRqqt3/assignment_loss‚2.5675741587624406Ù,ŠVXÇ Ò 2loss/SRqqt2/assignment_loss‚2.3502155523146353 1loss/FBt1/assignment_loss‚0.33672770426040916 _runtime‚1018.898746824 .loss/FRt3/assignment_loss‚5.42288925008076 2loss/SRqqt4/assignment_loss‚2.2273596860468388 /loss/FRt1/detection_loss‚1.9578814927547696 .loss/FRt2/detection_loss‚2.658751839815184 /loss/FRt4/detection_loss‚2.2744767974162925 1loss/SRqqt3/detection_loss‚2.7132514227520335 0loss/SRqqt4/detection_loss‚1.757147935398838 &loss/total_loss‚61.281267651865065 0loss/FBt2/assignment_loss‚0.5860430779866874 /loss/FBt3/assignment_loss‚0.725798743243594 .loss/FRt3/detection_loss‚2.665621881077929 trainer/global_step‚1349 _step‚110 /loss/FRt4/assignment_loss‚4.678150144116632 /loss/FBt1/detection_loss‚0.8871377822402114 .loss/FBt3/detection_loss‚4.462595891207457 .loss/FBt4/detection_loss‚4.379087708890438 # _timestamp‚1.78842933419408e+09 2loss/SRqqt3/assignment_loss‚2.5675741587624406 /loss/FRt2/assignment_loss‚4.789680123329163 0loss/FBt4/assignment_loss‚0.6338688023388386 1loss/SRqqt1/detection_loss‚1.2151799519824518 1loss/SRqqt2/detection_loss‚2.5837761000279458 2loss/SRqqt1/assignment_loss‚1.9178226117963915 .loss/FBt2/detection_loss‚4.545104891061783 epoch‚1 0loss/FRt1/assignment_loss‚2.9451241029633417nu0r÷žÈ ‚(:“ —ˆåÔÉýë gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚139.478' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.895600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ZVÉ ‚(:’ —ˆåÔãþ‘ disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0$ disk.nvme0n1.out‚5379.9169921875 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚0& disk.nvme0c0n1.out‚5379.9169921875 disk.drbd1777.in‚0 network.recv‚433907$ disk./.usageGB‚1099.719326019287 disk.dm-1.out‚ 42.515625 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 42.515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 55.4951171875 disk.sda3.out‚ 42.515625! proc.memory.rssMB‚ 2862.515625) proc.memory.percent‚17.47140884399414 disk.rbd4.out‚0 disk.dm-0.in‚0 disk.sda.in‚0 disk.sda2.in‚0 cpu‚25.328773499972236 network.sent‚ 1.893479e+06 disk.sda3.in‚0# memory_percent‚94.1582202911377! disk.dm-0.out‚5379.9169921875 disk.dm-1.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda2.out‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 957.1171875( disk./.usagePercent‚73.8030866951287 disk.nvme0c0n1.in‚0 proc.cpu.threads‚17±Øå“Ê ‚(:ˆ ¦ˆåÔÿž…Ž gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.8658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03=M Ë ‚(:• ¦ˆåÔñ™© network.recv‚439394 disk.rbd1.in‚0 disk.rbd4.in‚0 cpu‚25.324870548417756 disk.dm-0.in‚0 network.sent‚ 2.000768e+06% disk.nvme0n1.out‚5380.60400390625 disk.sda3.out‚43.3125 disk.drbd1777.in‚0 disk.rbd4.out‚0 disk.sda.out‚43.3125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0$ memory_percent‚94.15767192840576) proc.memory.percent‚17.47143268585205 disk.rbd2.in‚0" disk.dm-0.out‚5380.60400390625 disk.sda1.out‚0) disk./.usagePercent‚73.80308336703075 disk.rbd1.out‚ 0.61328125 disk.nvme0c0n1.in‚0 disk.sda1.in‚0% disk./.usageGB‚1099.7192764282227 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 57.0693359375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 957.20703125 disk.dm-1.out‚43.3125' disk.nvme0c0n1.out‚5380.60400390625 disk.sda2.in‚0# proc.memory.rssMB‚ 2862.51953125 proc.cpu.threads‚17‘˜• ’Ì ‚(:‡ µˆåÔ÷¬× gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.975' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.595 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>~*Í ‚(:Ÿ µˆåÔ¢Ÿ¬‘( proc.memory.availableMB‚ 956.97265625) proc.memory.percent‚17.47143268585205 disk.sda2.out‚0 disk.sda.out‚ 44.00390625 disk.sda1.out‚0 network.sent‚ 2.006992e+06 network.recv‚443919) disk./.usagePercent‚73.80307978292528 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚0# proc.memory.rssMB‚ 2862.51953125' disk.nvme0c0n1.out‚5479.72412109375 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 44.00390625 disk.rbd1.in‚0 cpu‚25.32098900543095$ disk./.usageGB‚1099.719223022461 disk.sda3.out‚ 44.00390625 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0% disk.nvme0n1.out‚5479.72412109375 disk.rbd1.out‚ 0.61328125 disk.nvme0c0n1.in‚0 disk.rbd2.out‚ 57.1787109375 disk.sda3.in‚0" disk.dm-0.out‚5479.72412109375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd4.in‚0 disk.nvme0n1.in‚0$ memory_percent‚94.15910243988037ÉuË2Î - _wandbruntime‚1054  _runtime‚1054!üh§ÇÏ Á # _timestamp‚1.78842936930937e+09 _runtime‚1054.014128942 _step‚111 ' lr-AdamW/pg1‚0.00022645235395610857 ' lr-AdamW/pg2‚0.00022645235395610857 trainer/global_step‚1399ï›ÖkËÐ Å trainer/global_step‚1399 # _timestamp‚1.78842936930937e+09 _runtime‚1054.014128942 _step‚111 ' lr-AdamW/pg1‚0.00022645235395610857 ' lr-AdamW/pg2‚0.00022645235395610857oQù=2Ñ - _wandbruntime‚1054  _runtime‚1054µâ°àÇ'Ò ÁO TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80271034932027 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1171027982336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81231440684544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚1399 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20310776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _step‚112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1229951413248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81231458687488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20310774 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82401812305408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81231458687488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26405876 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82402486669824 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79104990059520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20310776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79104973020160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81231440684544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18606816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18607263 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26385840ØM=ÙUÓ ÅO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46696614 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20310776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79104973020160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81231440684544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18606816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18607263 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26385840 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80271034932027 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1171027982336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81231440684544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚1399 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20310776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _step‚112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1229951413248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81231458687488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20310774 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82401812305408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81231458687488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26405876 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82402486669824 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79104990059520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79159894190080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26424356 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11858822 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884293699950554e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1171847423488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46749148 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6760631 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20324792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚1399 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1171191048704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81287793723392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1230969280000 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1171847423488 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90010283329024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20324792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46729084 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18619896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1166878346831 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚113 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46729084 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46749148 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80326755497551 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81287811726336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26424356 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1230797279744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚1054.700394274 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26404294 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1167532784787 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1171191048704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82458984772096 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90010479269376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18619451 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20324790 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81287793723392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6761074 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088¬ö<üÕ&Ö ÏM _runtime‚1054.700394274 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26404294 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1167532784787 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1171191048704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82458984772096 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90010479269376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18619451 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20324790 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81287793723392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6761074 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88779509989376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82459659149824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79159894190080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26424356 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11858822 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884293699950554e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1171847423488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46749148 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6760631 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20324792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚1399 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1171191048704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81287793723392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1230969280000 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1171847423488 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90010283329024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20324792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46729084 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18619896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1166878346831 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚113 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46729084 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46749148 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80326755497551 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81287811726336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26424356 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1230797279744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430q¹ë˨2× - _wandbruntime‚1054  _runtime‚1054µïË,UØ Ï /loss/FBt4/assignment_loss‚0.604048010375752 /loss/FRt3/detection_loss‚3.1599117196523228 1loss/SRqqt1/detection_loss‚1.4846188032901602 1loss/SRqqt3/detection_loss‚2.8153559820992604 .loss/FBt2/detection_loss‚5.742376089096069 $ _timestamp‚1.788429369996924e+09 /loss/FRt1/assignment_loss‚2.471833684227683 /loss/FRt2/assignment_loss‚3.685698697566986 1loss/SRqqt1/assignment_loss‚1.920758753818901 1loss/SRqqt3/assignment_loss‚2.552137166261673 /loss/FRt4/assignment_loss‚4.212581483125686 /loss/FRt1/detection_loss‚1.8183404830369083 /loss/FBt3/detection_loss‚3.0846661112525244 .loss/FBt4/detection_loss‚3.645314597421222 &loss/total_loss‚57.464505722059045 trainer/global_step‚1399 /loss/FBt2/assignment_loss‚0.796784520149231 1loss/FBt3/assignment_loss‚0.46160846000348826 2loss/SRqqt4/assignment_loss‚2.0525123172535977 /loss/FRt4/detection_loss‚2.1544000470638274 _runtime‚1054.701562095 _step‚114 .loss/FRt3/assignment_loss‚5.14841352059291 2loss/SRqqt2/assignment_loss‚2.0173842406019253 .loss/FBt1/detection_loss‚0.978427132256445 1loss/FBt1/assignment_loss‚0.21651893434757857 /loss/FRt2/detection_loss‚2.5291817563772203 1loss/SRqqt2/detection_loss‚2.0691816952634365 1loss/SRqqt4/detection_loss‚1.8424515169242333 epoch‚1¯ñlYYÙ Ó .loss/FRt3/assignment_loss‚5.14841352059291 2loss/SRqqt2/assignment_loss‚2.0173842406019253 .loss/FBt1/detection_loss‚0.978427132256445 1loss/FBt1/assignment_loss‚0.21651893434757857 /loss/FRt2/detection_loss‚2.5291817563772203 1loss/SRqqt2/detection_loss‚2.0691816952634365 1loss/SRqqt4/detection_loss‚1.8424515169242333 epoch‚1 /loss/FBt4/assignment_loss‚0.604048010375752 /loss/FRt3/detection_loss‚3.1599117196523228 1loss/SRqqt1/detection_loss‚1.4846188032901602 1loss/SRqqt3/detection_loss‚2.8153559820992604 .loss/FBt2/detection_loss‚5.742376089096069 $ _timestamp‚1.788429369996924e+09 /loss/FRt1/assignment_loss‚2.471833684227683 /loss/FRt2/assignment_loss‚3.685698697566986 1loss/SRqqt1/assignment_loss‚1.920758753818901 1loss/SRqqt3/assignment_loss‚2.552137166261673 /loss/FRt4/assignment_loss‚4.212581483125686 /loss/FRt1/detection_loss‚1.8183404830369083 /loss/FBt3/detection_loss‚3.0846661112525244 .loss/FBt4/detection_loss‚3.645314597421222 &loss/total_loss‚57.464505722059045 trainer/global_step‚1399 /loss/FBt2/assignment_loss‚0.796784520149231 1loss/FBt3/assignment_loss‚0.46160846000348826 2loss/SRqqt4/assignment_loss‚2.0525123172535977 /loss/FRt4/detection_loss‚2.1544000470638274 _runtime‚1054.701562095 _step‚114r‹¯nÞžÚ ‚(:“ ĈåÔÇ›Ý gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.471' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.294199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆÈà’&Û ‚(:› ĈåÔ–ý disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd4.in‚0% disk./.usageGB‚1099.7192611694336 disk.sda3.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚5579.287109375( proc.memory.availableMB‚ 956.89453125) proc.memory.percent‚17.47143268585205 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.rbd4.out‚0 disk.sda2.out‚0 disk.sda.out‚ 44.76953125 disk.sda1.in‚0 disk.rbd1.out‚ 0.61328125% disk.nvme0c0n1.out‚5579.287109375 disk.sda2.in‚0 disk.rbd2.in‚0 network.recv‚447879 disk.sda.in‚0 disk.sda1.out‚0 network.sent‚ 2.012495e+06 disk.nvme0n1.in‚0 disk.sda3.out‚ 44.76953125 disk.drbd1777.in‚0 disk.dm-1.in‚0$ memory_percent‚94.15957927703857 cpu‚25.317316526663785 disk.rbd1.in‚0 disk.rbd2.out‚ 57.1787109375 disk.drbd1777.out‚0 disk.dm-1.out‚ 44.76953125# proc.memory.rssMB‚ 2862.51953125) disk./.usagePercent‚73.80308234300061# disk.nvme0n1.out‚5579.287109375 disk.nvme0c0n1.in‚0žV퉞Ü ‚(:“ ÓˆåÔÕöÞ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.692' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.138400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬]$Ý ‚(:Ž ÓˆåÔµý disk.rbd4.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0# disk.nvme0n1.out‚5678.404296875 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚45.3125 disk.rbd1.in‚0$ memory_percent‚94.15926933288574 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.nvme0n1.in‚0' proc.memory.availableMB‚ 956.9453125 cpu‚25.31370141195928 disk.rbd3.in‚0 disk.dm-0.out‚5678.404296875 disk.sda.out‚45.3125 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚45.3125 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2862.51953125% disk.nvme0c0n1.out‚5678.404296875 network.sent‚ 2.121048e+06 network.recv‚454478 disk.drbd1777.in‚0 disk.sda.in‚0 disk.sda1.out‚0) disk./.usagePercent‚73.80307875889514% disk./.usageGB‚1099.7192077636719 disk.rbd1.out‚ 0.61328125) proc.memory.percent‚17.47143268585205 disk.rbd2.out‚57.34716796875 disk.drbd1777.out‚0 disk.dm-0.in‚0º#ëS2Þ - _wandbruntime‚1089  _runtime‚1089yØb Æß À ' lr-AdamW/pg1‚0.00022635087838944758 ' lr-AdamW/pg2‚0.00022635087838944758 trainer/global_step‚1449 # _timestamp‚1.78842940462917e+09 _runtime‚ 1089.33391032 _step‚115øQ×Êà Ä ' lr-AdamW/pg1‚0.00022635087838944758 ' lr-AdamW/pg2‚0.00022635087838944758 trainer/global_step‚1449 # _timestamp‚1.78842940462917e+09 _runtime‚ 1089.33391032 _step‚115s™íך2á - _wandbruntime‚1089  _runtime‚1089‹"ó â ÁO trainer/global_step‚1449 _step‚116 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1212013788160 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6984997 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚91800396134912 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1206897251275 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21011574 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21011576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12259054 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85261124418048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1273088425472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1207551689023 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884294046300492e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1212013788160 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93073484560384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48343504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85260450053632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84049110629888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19244485 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83057076799435 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85261124418048 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85260450053632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12259052 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21011576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1211357426688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84049092626944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27331928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83057748276543 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1272918509056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414/ÝdË'ã ÅO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1211357426688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚1089.335118257 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27311892 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6985431 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81850196587520 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81850179548160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169916416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48323466 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84049092626944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91800372194816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84049110629888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27311892 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19244049 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21011574 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85261124418048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1273088425472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1207551689023 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884294046300492e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1212013788160 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93073484560384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48343504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85260450053632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84049110629888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19244485 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83057076799435 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85261124418048 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85260450053632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12259052 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21011576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1211357426688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84049092626944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27331928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83057748276543 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1272918509056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚1449 _step‚116 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1212013788160 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6984997 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚91800396134912 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1206897251275 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21011574 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21011576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12259054 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48355936 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27330346 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19257199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21025590 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48376000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1212833229312 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21025592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27350408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84105463668736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1212176854528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚91862022932480 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81905083678720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48355936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91862046872576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1273939154944 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1208368124195 _runtime‚1089.995788987 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81905100718080 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21025592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167805952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1273771348992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21025590 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83112797364959 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48376000 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84105463668736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191746048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85317622520320 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93135986027520 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83113468842275 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12267220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85318296898048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19256757 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12267222 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6989977 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1207713686239 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27350408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84105445665792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84105445665792 _step‚117 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27330346 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0˜Yp;æ ÏM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21025590 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83112797364959 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48376000 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84105463668736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191746048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85317622520320 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93135986027520 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83113468842275 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12267220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85318296898048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19256757 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12267222 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6989977 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1207713686239 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27350408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84105445665792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84105445665792 _step‚117 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27330346 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48355936 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 zd™hšNADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27330346 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19257199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21025590 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48376000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1212833229312 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21025592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27350408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84105463668736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1212176854528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚91862022932480 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81905083678720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48355936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91862046872576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1273939154944 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1208368124195 _runtime‚1089.995788987 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81905100718080 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21025592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167805952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1273771348992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728uqå2ç - _wandbruntime‚1089  _runtime‚1089k#ŒœUè Ï 0loss/FBt1/assignment_loss‚0.1813243201339458 /loss/FRt3/detection_loss‚2.9224592316895723 trainer/global_step‚1449 0loss/FBt3/assignment_loss‚0.6414516419172287 /loss/FRt3/assignment_loss‚5.184592850506306 2loss/SRqqt1/assignment_loss‚1.7594117268043405 .loss/FBt2/detection_loss‚4.126286268234253 0loss/FBt4/assignment_loss‚0.8463243097066879 /loss/FBt4/detection_loss‚2.5959902306397757 $loss/total_loss‚54.6393848126459 /loss/FRt4/assignment_loss‚4.114727921783924 /loss/FRt4/detection_loss‚2.0749472895016274 /loss/FBt3/detection_loss‚3.0950092905097537 _runtime‚1089.997179999 /loss/FRt1/detection_loss‚2.1049720346927643 1loss/SRqqt1/detection_loss‚1.4374334992950453 1loss/SRqqt2/detection_loss‚1.9793662410229445 epoch‚1 $ _timestamp‚1.788429405292225e+09 _step‚118 /loss/FRt1/assignment_loss‚3.008558780563121 0loss/FBt2/assignment_loss‚0.5760211196156888 1loss/SRqqt3/detection_loss‚2.9341075086593627 0loss/SRqqt4/detection_loss‚2.019490210339427 /loss/FBt1/detection_loss‚0.6896098977145907 .loss/FRt2/detection_loss‚2.410832523057858 /loss/FRt2/assignment_loss‚4.081816958884398 1loss/SRqqt2/assignment_loss‚1.860680654955407 2loss/SRqqt3/assignment_loss‚2.2012452590465545 2loss/SRqqt4/assignment_loss‚1.7927250433713198Ü$°§Yé Ó 2loss/SRqqt1/assignment_loss‚1.7594117268043405 .loss/FBt2/detection_loss‚4.126286268234253 0loss/FBt4/assignment_loss‚0.8463243097066879 /loss/FBt4/detection_loss‚2.5959902306397757 $loss/total_loss‚54.6393848126459 /loss/FRt4/assignment_loss‚4.114727921783924 /loss/FRt4/detection_loss‚2.0749472895016274 /loss/FBt3/detection_loss‚3.0950092905097537 _runtime‚1089.997179999 /loss/FRt1/detection_loss‚2.1049720346927643 1loss/SRqqt1/detection_loss‚1.4374334992950453 1loss/SRqqt2/detection_loss‚1.9793662410229445 epoch‚1 $ _timestamp‚1.788429405292225e+09 _step‚118 /loss/FRt1/assignment_loss‚3.008558780563121 0loss/FBt2/assignment_loss‚0.5760211196156888 1loss/SRqqt3/detection_loss‚2.9341075086593627 0loss/SRqqt4/detection_loss‚2.019490210339427 /loss/FBt1/detection_loss‚0.6896098977145907 .loss/FRt2/detection_loss‚2.410832523057858 /loss/FRt2/assignment_loss‚4.081816958884398 1loss/SRqqt2/assignment_loss‚1.860680654955407 2loss/SRqqt3/assignment_loss‚2.2012452590465545 2loss/SRqqt4/assignment_loss‚1.7927250433713198 0loss/FBt1/assignment_loss‚0.1813243201339458 /loss/FRt3/detection_loss‚2.9224592316895723 trainer/global_step‚1449 0loss/FBt3/assignment_loss‚0.6414516419172287 /loss/FRt3/assignment_loss‚5.184592850506306v0Ñ“ê ‚(:ˆ âˆåÔðžð gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.623' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ %Ê,ë ‚(:¡ âˆåÔð€µ disk.rbd0.out‚0 disk.rbd2.out‚57.34716796875% disk./.usageGB‚1099.7190284729004 disk.sda2.in‚0 disk.rbd2.in‚0# memory_percent‚94.1626787185669 disk.dm-1.out‚ 45.81640625 disk.sda.out‚ 45.81640625) disk./.usagePercent‚73.80306672654102 disk.drbd1777.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2862.51953125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚5777.67236328125 disk.rbd1.in‚0 disk.sda2.out‚0 cpu‚25.309931196917603 network.recv‚458156 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.dm-0.in‚0' disk.nvme0c0n1.out‚5777.67236328125 disk.rbd0.in‚0 disk.sda3.out‚ 45.81640625( proc.memory.availableMB‚ 956.38671875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚0) proc.memory.percent‚17.47143268585205 proc.cpu.threads‚17" disk.dm-0.out‚5777.67236328125 network.sent‚ 2.126255e+06 disk.rbd1.out‚ 0.61328125 disk.sda.in‚0 disk.sda1.in‚0¤_ÎÝì ‚(:’ ñˆåÔëÑÎ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.76' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.351999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õDfè(í ‚(: ñˆåÔ³óù) disk./.usagePercent‚73.80306160639034 disk.rbd1.in‚0 disk.dm-0.in‚0) proc.memory.percent‚17.47143268585205 cpu‚25.306474075083056 disk.rbd0.in‚0 disk.drbd1777.in‚0% proc.memory.availableMB‚ 955.84375 network.recv‚464621 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚57.47216796875$ disk./.usageGB‚1099.718952178955 disk.drbd1777.out‚0 disk.rbd3.out‚0# memory_percent‚94.1659927368164 disk.rbd1.out‚ 0.61328125 network.sent‚ 2.234937e+06 disk.sda.in‚0 disk.dm-1.out‚ 46.29296875 disk.rbd4.out‚0" disk.dm-0.out‚5777.97705078125 disk.dm-1.in‚0% disk.nvme0n1.out‚5777.97705078125 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 46.29296875 disk.sda1.out‚0' disk.nvme0c0n1.out‚5777.97705078125 disk.sda2.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2862.51953125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 46.29296875 disk.rbd4.in‚0årÃ}“î ‚(:ˆ €‰åÔüÀí gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.6678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÞÓ+ï ‚(:  €‰åÔéÚ£ disk.sda3.out‚ 46.84765625 disk.drbd1777.in‚0 disk.sda.in‚0 disk.nvme0c0n1.in‚0' proc.memory.availableMB‚ 956.2578125 network.sent‚ 2.240345e+06 disk.rbd4.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 network.recv‚468344 disk.rbd4.in‚0$ memory_percent‚94.16346549987793# proc.memory.rssMB‚ 2862.51953125$ disk./.usageGB‚1099.719009399414 disk.rbd3.out‚0) proc.memory.percent‚17.47143268585205 cpu‚25.303162294565272 disk.rbd3.in‚0 disk.dm-0.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 0.61328125 disk.nvme0n1.in‚0) disk./.usagePercent‚73.80306544650335" disk.dm-0.out‚5877.24951171875 disk.sda.out‚ 46.84765625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 46.84765625 disk.sda3.in‚0 disk.drbd1777.out‚0% disk.nvme0n1.out‚5877.24951171875' disk.nvme0c0n1.out‚5877.24951171875 disk.rbd0.out‚0 disk.rbd2.out‚59.06201171875Aƒz2ð - _wandbruntime‚1125  _runtime‚1125`÷žªÇñ Á % _timestamp‚1.7884294403561597e+09 _runtime‚1125.060870724 _step‚119 & lr-AdamW/pg1‚0.0002262258455656175 & lr-AdamW/pg2‚0.0002262258455656175 trainer/global_step‚1499”|×pËò Å & lr-AdamW/pg1‚0.0002262258455656175 & lr-AdamW/pg2‚0.0002262258455656175 trainer/global_step‚1499 % _timestamp‚1.7884294403561597e+09 _runtime‚1125.060870724 _step‚119w5ö02ó - _wandbruntime‚1125  _runtime‚1125 | Æ'ô ÀO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1252999593984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1316115444736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49970356 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21712374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96198858800640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88119087801856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21712376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚1125.061757876 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28237944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21712374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28237944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12667454 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94882909078016 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7213775 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28257980 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86866744569344 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚85843118666843 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1315949722624 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86866762572288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86866762572288 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49950318 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1252999593984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19881227 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788429440356844e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1247732590683 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21712376 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88119087801856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86866744569344 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1252343232512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84595403115520 trainer/global_step‚1499 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚85843790143951ÙjÊ'õ ÄO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49970356 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21712374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96198858800640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88119087801856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21712376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚1125.061757876 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28237944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21712374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28237944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12667454 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94882909078016 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7213775 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28257980 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86866744569344 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚85843118666843 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1315949722624 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86866762572288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86866762572288 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49950318 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1252999593984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19881227 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788429440356844e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1247732590683 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21712376 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88119087801856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86866744569344 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1252343232512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84595403115520 trainer/global_step‚1499 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚85843790143951 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1252999593984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1316115444736x©o±ï2ö - _wandbruntime‚1125  _runtime‚11259x%J÷ ÊM M>Devi& WÅÂ&ceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19894051 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49982788 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84650307246080 trainer/global_step‚1499 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94944583755776 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50002852 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19894490 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚85898839232367 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21726392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86923097608192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1316977635840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86923115611136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84650290206720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50002852 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12675620 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28256398 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94944559815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7218868 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1253819035136 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1248549025647 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28276460 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88176934646272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96261380131328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253162660352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1249203463603 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚1125.730842677{ÌM¦Ô&ø ÎM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1316977635840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86923115611136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84650290206720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50002852 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12675620 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28256398 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94944559815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7218868 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1253819035136 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1248549025647 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28276460 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88176934646272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96261380131328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253162660352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1249203463603 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚1125.730842677 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19894051 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49982788 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84650307246080 trainer/global_step‚1499 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94944583755776 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50002852 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19894490 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚85898839232367 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21726392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86923097608192yLÃDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚420 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚1160.410577582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89684396511744 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90977725550080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚87340592604160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29184032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22413176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚422 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1289222367839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51577170 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90977725550080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29163996 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7443119 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7442699 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90978399914496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97965445961216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99324368066560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29163996 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89684396511744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20518973 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13075854 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1288567930091 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22413174 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1293985399808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179176448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884294757056756e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22413174 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89684414514688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1293985399808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1359077341696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155236352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024•PéË'…ÅO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚1160.410577582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89684396511744 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90977725550080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚87340592604160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29184032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22413176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚422 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1289222367839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51577170 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90977725550080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29163996 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7443119 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7442699 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90978399914496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97965445961216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99324368066560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29163996 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89684396511744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20518973 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13075854 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1288567930091 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22413174 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1293985399808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179176448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884294757056756e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22413174 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89684414514688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1293985399808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1359077341696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155236352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97965469901312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1293329038336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13075852 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88629160534251 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 _step‚124 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1293329038336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89684414514688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20518551 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22413176 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90978399914496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚420 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0|í«ë2†- _wandbruntime‚1161  _runtime‚1161vªnÑ&‡ËM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7447206 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7447647 % _timestamp‚1.7884294763711412e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29202512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22427190 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20531669 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91035572394496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89740749550592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51629704 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51609640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89740749550592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 _runtime‚1161.076233662 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1294804840960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89740767553536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1294804840960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98027096698880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1359762685952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98027120638976 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13084022 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89740767553536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1289384365055 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1359940977664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22427192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22427192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29182450 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1294148466176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13084020 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20531226 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 trainer/global_step‚1549 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99386859384832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88685552577091 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87395496734720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29182450 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88684881099775 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87395513774080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51629704 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99387061616640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430»wÄ#ˆÏM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1289384365055 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1359940977664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22427192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22427192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29182450 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1294148466176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13084020 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20531226 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 trainer/global_step‚1549 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99386859384832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88685552577091 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87395496734720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29182450 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88684881099775 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87395513774080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51629704 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99387061616640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7447206 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7447647 % _timestamp‚1.7884294763711412e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29202512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22427190 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20531669 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91035572394496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89740749550592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51629704 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51609640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89740749550592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 _runtime‚1161.076233662 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1294804840960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89740767553536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1294804840960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98027096698880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1359762685952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98027120638976 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13084022 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89740767553536}&&P2‰- _wandbruntime‚1161  _runtime‚1161<Å›ZŠÔ /loss/FRt4/detection_loss‚2.4803286991840183 trainer/global_step‚1549 0loss/FRt2/assignment_loss‚3.7799646029105554 2loss/SRqqt2/assignment_loss‚2.1150448251781717 0loss/FBt1/assignment_loss‚0.3248301909418479 0loss/SRqqt1/detection_loss‚1.492723941190602 1loss/SRqqt3/detection_loss‚3.0277002923763714 2loss/SRqqt1/assignment_loss‚1.7062819797086388 2loss/SRqqt4/assignment_loss‚1.8422372762749835 0loss/SRqqt4/detection_loss‚2.201583676221894 0loss/FRt3/assignment_loss‚4.7191170374552405 0loss/FBt2/assignment_loss‚0.5810381594845718 1loss/SRqqt2/detection_loss‚1.9738827613847596 %loss/total_loss‚55.82695772669674 /loss/FRt4/assignment_loss‚4.075458357500476 2loss/SRqqt3/assignment_loss‚2.3763498022006107 0loss/FBt3/assignment_loss‚0.3653265904220196 0loss/FBt4/assignment_loss‚0.8543560604254404 _runtime‚1161.077535377 epoch‚1 0loss/FRt1/assignment_loss‚2.7657789109008655 /loss/FRt1/detection_loss‚1.8594729797914624 /loss/FBt1/detection_loss‚0.6739563709686778 /loss/FBt4/detection_loss‚2.4121609568595885 .loss/FRt2/detection_loss‚2.529263056241549 /loss/FBt2/detection_loss‚3.8720657655170987 /loss/FBt3/detection_loss‚3.6913019683625965 $ _timestamp‚1.788429476372924e+09 _step‚126 /loss/FRt3/detection_loss‚4.1067334651947025¤°C ^‹Ø 0loss/FRt3/assignment_loss‚4.7191170374552405 0loss/FBt2/assignment_loss‚0.5810381594845718 1loss/SRqqt2/detection_loss‚1.9738827613847596 %loss/total_loss‚55.82695772669674 /loss/FRt4/assignment_loss‚4.075458357500476 2loss/SRqqt3/assignment_loss‚2.3763498022006107 0loss/FBt3/assignment_loss‚0.3653265904220196 0loss/FBt4/assignment_loss‚0.8543560604254404 _runtime‚1161.077535377 epoch‚1 0loss/FRt1/assignment_loss‚2.7657789109008655 /loss/FRt1/detection_loss‚1.8594729797914624 /loss/FBt1/detection_loss‚0.6739563709686778 /loss/FBt4/detection_loss‚2.4121609568595885 .loss/FRt2/detection_loss‚2.529263056241549 /loss/FBt2/detection_loss‚3.8720657655170987 /loss/FBt3/detection_loss‚3.6913019683625965 $ _timestamp‚1.788429476372924e+09 _step‚126 /loss/FRt3/detection_loss‚4.1067334651947025 /loss/FRt4/detection_loss‚2.4803286991840183 trainer/global_step‚1549 0loss/FRt2/assignment_loss‚3.7799646029105554 2loss/SRqqt2/assignment_loss‚2.1150448251781717 0loss/FBt1/assignment_loss‚0.3248301909418479 0loss/SRqqt1/detection_loss‚1.492723941190602 1loss/SRqqt3/detection_loss‚3.0277002923763714 2loss/SRqqt1/assignment_loss‚1.7062819797086388 2loss/SRqqt4/assignment_loss‚1.8422372762749835 0loss/SRqqt4/detection_loss‚2.201583676221894~lÜë“Œ‚(:ˆ ­‰åÔñ߆Ž gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÆ’è+‚(:  ­‰åÔ²·‘ network.recv‚481795 disk.rbd4.out‚0 disk.dm-0.in‚0" disk.dm-0.out‚6175.72705078125 disk.sda1.out‚0 disk.nvme0c0n1.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 48.79296875% disk./.usageGB‚1099.7195014953613 disk.sda.in‚0 disk.rbd1.out‚ 0.61328125' disk.nvme0c0n1.out‚6175.72705078125 disk.rbd0.in‚0 disk.sda3.in‚0) disk./.usagePercent‚73.80309847147527 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0$ memory_percent‚94.16418075561523 proc.cpu.threads‚17 disk.rbd1.in‚0& proc.memory.availableMB‚ 956.140625 cpu‚25.293586547049014 disk.dm-1.in‚0% disk.nvme0n1.out‚6175.72705078125 disk.drbd1777.out‚0 disk.rbd3.in‚0) proc.memory.percent‚17.47143268585205 disk.sda.out‚ 48.79296875 disk.dm-1.out‚ 48.79296875 disk.sda2.in‚0 disk.rbd2.out‚59.22216796875# proc.memory.rssMB‚ 2862.51953125 network.sent‚ 2.359351e+06 disk.rbd4.in‚0 disk.nvme0n1.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0ŒÁXꞎ‚(:“ ¼‰åÔŽïß gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.166' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.833199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£!àt‚(:Š ¼‰åÔÒ˜‘ network.sent‚ 2.467189e+06 proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0" disk.nvme0n1.out‚ 6176.33203125 disk.rbd1.out‚ 0.61328125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.memory.availableMB‚7.5 network.recv‚487632 disk.rbd4.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 49.48046875 disk.dm-1.out‚ 49.48046875% disk./.usageGB‚1099.7193756103516 disk.dm-0.in‚0 disk.dm-0.out‚ 6176.33203125 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0$ disk.nvme0c0n1.out‚ 6176.33203125) disk./.usagePercent‚73.80309002322664 disk.sda.out‚ 49.48046875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd2.out‚ 59.265625) proc.memory.percent‚17.47143268585205 disk.rbd4.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2862.51953125 cpu‚25.290579974581377# memory_percent‚99.9542236328125 disk.sda3.in‚0÷#+d2- _wandbruntime‚1195  _runtime‚1195¯LGÉ‘Ã trainer/global_step‚1599 % _timestamp‚1.7884295112821653e+09 _runtime‚1195.986905817 _step‚127 ' lr-AdamW/pg1‚0.00022590521727692953 ' lr-AdamW/pg2‚0.00022590521727692953ÙZÍ’Ç ' lr-AdamW/pg1‚0.00022590521727692953 ' lr-AdamW/pg2‚0.00022590521727692953 trainer/global_step‚1599 % _timestamp‚1.7884295112821653e+09 _runtime‚1195.986905817 _step‚127ƒT².2“- _wandbruntime‚1195  _runtime‚1195R‡ŒeË'”ÅO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13484252 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚1599 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1334971205632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93836363298304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92502048454144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30110084 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1334971205632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7671900 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91415202401659 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1330057707247 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23113974 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23113976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7671449 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21155701 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93837037662720 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 _step‚128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30110084 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93837037662720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92502066457088 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚91415873878767 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93836363298304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102450009562624 % _timestamp‚1.7884295112829125e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1334314844160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0€’ ÍÐ'•ÊO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21156154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101047982844416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚1195.987843234 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1330057707247 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23113974 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23113976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7671449 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21155701 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93837037662720 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 _step‚128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30110084 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93837037662720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92502066457088 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚91415873878767 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93836363298304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102450009562624 % _timestamp‚1.7884295112829125e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1334314844160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13484252 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚1599 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1334971205632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93836363298304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92502048454144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30110084 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1334971205632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7671900 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91415202401659 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53256556 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1335134272000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91470922967183 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7676390 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92558419495936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1335790646784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23127992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30128564 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101109657522176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101109633582080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1402715600384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13492420 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90140703262720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23127990 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.larg`ð)e_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92558419495936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92558401492992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90140720302080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1330219704463 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13492422 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102512530442752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚1196.649311475 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21168370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30128564 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21168812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30108502 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91471594444499 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93893535764992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102512349182464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1330874142419 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1335134272000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884295119439685e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23127992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30128564 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101109657522176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101109633582080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1402715600384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13492420 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90140703262720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23127990 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92558419495936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92558401492992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90140720302080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1330219704463 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13492422 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102512530442752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚1196.649311475 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21168370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30128564 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21168812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30108502 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91471594444499 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93893535764992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102512349182464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1330874142419 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1335134272000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884295119439685e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53256556 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1335134272000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91470922967183 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7676390 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92558419495936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1335790646784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 BŸ‚(:“ Ú‰åÔò»% disk.nvme0c0n1.out‚6374.654296875 disk.drbd1777.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.dm-0.in‚0 disk.sda3.in‚0) proc.memory.percent‚17.47143268585205 network.sent‚ 2.581171e+06 disk.sda.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0$ memory_percent‚99.95477199554443 disk.sda.out‚ 50.5703125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 60.87109375 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.dm-1.out‚ 50.5703125 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 50.5703125# proc.memory.rssMB‚ 2862.51953125 cpu‚25.28480864006508) disk./.usagePercent‚73.80315581716295% disk./.usageGB‚1099.7203559875488 disk.rbd3.in‚0# disk.nvme0n1.out‚6374.654296875 disk.rbd2.in‚0& proc.memory.availableMB‚ 7.41015625 network.recv‚497757 disk.dm-0.out‚6374.654296875 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.rbd1.out‚ 0.61328125ÀÙ_Dž ‚(:“ é‰åÔ§¢ð gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.523' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.104599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{(†É!¡‚(:– é‰åÔ’¥% proc.memory.availableMB‚ 7.1328125 proc.cpu.threads‚17 disk.rbd1.out‚ 0.61328125 disk.sda3.out‚ 51.12109375 disk.rbd3.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.47143268585205% disk.nvme0c0n1.out‚6473.701171875 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda.in‚0) disk./.usagePercent‚73.80315197704994 disk.rbd0.in‚0 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 51.12109375$ memory_percent‚99.95646476745605# proc.memory.rssMB‚ 2862.51953125 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 51.12109375 network.sent‚ 2.587395e+06 disk.rbd3.out‚0 disk.dm-1.in‚0 cpu‚25.281798775737766 disk.dm-0.out‚6473.701171875% disk./.usageGB‚1099.7202987670898 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0# disk.nvme0n1.out‚6473.701171875 network.recv‚502467 disk.rbd2.out‚ 60.87109375 disk.rbd4.in‚0 disk.sda1.in‚0ŽÅ<š2¢- _wandbruntime‚1231  _runtime‚1231¼²¸«É£Ã ' lr-AdamW/pg1‚0.00022570968863052637 ' lr-AdamW/pg2‚0.00022570968863052637 trainer/global_step‚1649 % _timestamp‚1.7884295468408074e+09 _runtime‚1231.545525483 _step‚131 ú§µÎ¤È ' lr-AdamW/pg1‚0.00022570968863052637 ' lr-AdamW/pg2‚0.00022570968863052637 trainer/global_step‚1649 % _timestamp‚1.7884295468408074e+09 _runtime‚1231.545525483 _step‚131ƒuV€2¥- _wandbruntime‚1231  _runtime‚1231Z„yÀy'¦ÅO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚95319718399488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13892652 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95319700396544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54850912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7898017 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104130543667712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104130519727616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚1649 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94201915746175 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23814774 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21791134 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1444942685184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31036136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31016100 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1375300649984 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96695001046528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1375957011456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1375957011456 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96695001046528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1375957011456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1375957011456 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚95319718399488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13892652 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95319700396544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54850912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7898017 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104130543667712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104130519727616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚1649 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94201915746175 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23814774 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21791134 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1444942685184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31036136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31016100 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1375300649984 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13900822 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1376776452608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104192194405376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31054616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23828790 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚92885926830080 _runtime‚1232.209822952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚95376053435392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96752847890944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104192170465280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95376071438336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31034554 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1376776452608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚133 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1376120077824 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94256964834591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1371055043871 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884295475044475e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1376120077824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 trainer/global_step‚1649 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1445792343552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚92885909790720 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23828792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31034554 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7902932 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7902481 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21803301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23828790 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105638158749184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31054616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54883408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94257636311907+"l÷Ú&ªÔM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31054616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54883408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94257636311907 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23828792 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96752173513216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13900822 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1376776452608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104192194405376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31054616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23828790 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚92885926830080 _runtime‚1232.209822952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚95376053435392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96752847890944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104192170465280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95376071438336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31034554 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1376776452608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚133 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1376120077824 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94256964834591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1371055043871 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884295475044475e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1376120077824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 trainer/global_step‚1649 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1445792343552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚92885909790720 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23828792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31034554 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7902932 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7902481 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21803301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23828790 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105638158749184…ó›#‚2«- _wandbruntime‚1232  _runtime‚1232¡3AZ¬Ô _runtime‚1232.211123162 /loss/FRt1/detection_loss‚1.9019939860230999 2loss/SRqqt2/assignment_loss‚1.9571778518812997 2loss/SRqqt3/assignment_loss‚2.4081952523659256 2loss/SRqqt4/assignment_loss‚1.8136644824235528 % _timestamp‚1.7884295475065014e+09 .loss/FRt3/detection_loss‚3.606449842453003 1loss/SRqqt4/detection_loss‚1.7803485796407417 _step‚134 /loss/FRt2/assignment_loss‚4.489426952893616 /loss/FRt4/assignment_loss‚4.719644600436801 .loss/FRt4/detection_loss‚2.174448326230049 1loss/SRqqt2/detection_loss‚1.9338949306734972 /loss/FBt1/detection_loss‚1.5340234992383073 .loss/FBt4/detection_loss‚2.623692049429967 1loss/FBt1/assignment_loss‚0.20236371702661782 1loss/SRqqt3/detection_loss‚2.3416459490271175 /loss/FBt3/detection_loss‚3.5854121074080467 0loss/FBt4/assignment_loss‚0.5954650434163901 2loss/SRqqt1/assignment_loss‚1.3271432872096154 0loss/FBt2/assignment_loss‚0.3279588597943075 1loss/SRqqt1/detection_loss‚1.4687238515750185 epoch‚1 /loss/FRt3/assignment_loss‚4.856861102581024 .loss/FRt2/detection_loss‚2.112684108194758 0loss/FBt3/assignment_loss‚0.3253177884028844 0loss/FRt1/assignment_loss‚2.7531045905330727 /loss/FBt2/detection_loss‚3.6301921010017395 &loss/total_loss‚54.469832859860446 trainer/global_step‚1649Y¦‚9­Ù .loss/FRt3/detection_loss‚3.606449842453003 1loss/SRqqt4/detection_loss‚1.7803485796407417 _step‚134 /loss/FRt2/assignment_loss‚4.489426952893616 /loss/FRt4/assignment_loss‚4.719644600436801 .loss/FRt4/detection_loss‚2.174448326230049 1loss/SRqqt2/detection_loss‚1.9338949306734972 /loss/FBt1/detection_loss‚1.5340234992383073 .loss/FBt4/detection_loss‚2.623692049429967 1loss/FBt1/assignment_loss‚0.20236371702661782 1loss/SRqqt3/detection_loss‚2.3416459490271175 /loss/FBt3/detection_loss‚3.5854121074080467 0loss/FBt4/assignment_loss‚0.5954650434163901 2loss/SRqqt1/assignment_loss‚1.3271432872096154 0loss/FBt2/assignment_loss‚0.3279588597943075 1loss/SRqqt1/detection_loss‚1.4687238515750185 epoch‚1 /loss/FRt3/assignment_loss‚4.856861102581024 .loss/FRt2/detection_loss‚2.112684108194758 0loss/FBt3/assignment_loss‚0.3253177884028844 0loss/FRt1/assignment_loss‚2.7531045905330727 /loss/FBt2/detection_loss‚3.6301921010017395 &loss/total_loss‚54.469832859860446 trainer/glïO[¥&obal_step‚1649 _runtime‚1232.211123162 /loss/FRt1/detection_loss‚1.9019939860230999 2loss/SRqqt2/assignment_loss‚1.9571778518812997 2loss/SRqqt3/assignment_loss‚2.4081952523659256 2loss/SRqqt4/assignment_loss‚1.8136644824235528 % _timestamp‚1.7884295475065014e+09†ýpÜs“®‚(:ˆ ø‰åԯ܄Ž gpu.0.gpu‚43.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05-–:'¯‚(:œ ø‰åÔð‰Ô disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2862.51953125 disk.sda3.out‚ 51.8671875" disk.dm-0.out‚6474.23583984375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd2.out‚61.03955078125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚6474.23583984375% disk./.usageGB‚1099.7203941345215 disk.rbd3.in‚0 disk.sda.out‚ 51.8671875 disk.dm-1.in‚0 disk.rbd2.in‚0$ memory_percent‚99.95534420013428 network.sent‚ 2.592809e+06 disk.rbd1.out‚ 0.61328125 disk.rbd0.out‚0 disk.sda1.in‚0) disk./.usagePercent‚73.80315837723829 disk.rbd4.out‚0& proc.memory.availableMB‚ 7.31640625 cpu‚25.27904775058416 network.recv‚506400 disk.sda3.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚6474.23583984375) proc.memory.percent‚17.47143268585205 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 51.8671875 disk.nvme0n1.in‚0 disk.rbd1.in‚0(›†ž°‚(:“ ‡ŠåÔÂ¥Þ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.786' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.757199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÅã+±‚(:  ‡ŠåÔ®†å disk.rbd1.out‚ 0.61328125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd4.out‚0 disk.dm-0.in‚0 disk.sda.out‚ 52.63671875$ memory_percent‚99.95381832122803 network.sent‚ 2.701365e+06) disk./.usagePercent‚73.80315453712527 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd4.in‚0 cpu‚25.276411680993192% disk.nvme0n1.out‚6573.34228515625 disk.sda2.in‚0 disk.sda3.out‚ 52.63671875 disk.drbd1777.in‚0 disk.drbd1777.out‚0% disk./.usageGB‚1099.7203369140625 disk.nvme0n1.in‚0 disk.nvme0c0n1.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2862.51953125 network.recv‚512729 disk.dm-1.out‚ 52.63671875' disk.nvme0c0n1.out‚6573.34228515625 disk.rbd2.out‚61.03955078125 disk.sda1.out‚0) proc.memory.percent‚17.47143268585205 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚6573.34228515625& proc.memory.availableMB‚ 7.56640625£Y÷2²- _wandbruntime‚1266  _runtime‚1266‡š÷îdzÁ _runtime‚1266.901770181 _step‚135 & lr-AdamW/pg1‚0.0002254907363499389 & lr-AdamW/pg2‚0.0002254907363499389 trainer/global_step‚1699 % _timestamp‚1.7884295821970036e+09:¹æÌ´Æ _runtime‚1266.901770181 _step‚135 & lr-AdamW/pg1‚0.0002254907363499389 & lr-AdamW/pg2‚0.0002254907363499389 trainer/global_step‚1699 % _timestamp‚1.7884295821970036e+09‡X˜—2µ- _wandbruntime‚1266  _runtime‚1266¼.Þ~Ë'¶ÅO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31962188 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14301052 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1411728386063 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1416942817280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22426995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108700995611136 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95576229227520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24515576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1416286455808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1411073948315 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99553638794752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1488115208192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1416942817280 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8125941 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107213080550912 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24515576 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24515574 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1487939000320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56477764 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14301054 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96987286136475 _step‚136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1416286455808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22426539 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99553638794752 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98137352338944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96987957613583 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98137370341888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107213056610816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31962188 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98137370341888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31942152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24515574 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108701195759104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0d)ç_Ð'·ÊO TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99553638794752 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98137352338944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96987957613583 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98137370341888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107213056610816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31962188 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98137370341888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31942152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24515574 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108701195759104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31962188 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14301052 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1411728386063 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1416942817280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22426995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108700995611136 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95576229227520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24515576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1416286455808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1411073948315 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99553638794752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1488115208192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1416942817280 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8125941 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107213080550912 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24515576 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24515574 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1487939000320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56477764 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14301054 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96987286136475 _step‚136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1416286455808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22426539ˆ g±û2¸- _wandbruntime‚1267  _runtime‚1267Qüý%8!¹ÏM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56510260 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99611485639168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22439229 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108763693703680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107274731288576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97043678179315 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22439674 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98193723380736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99610811261440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24529592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14309220 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24529590 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24529592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167805952 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56490196 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1412544821235 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14309222 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚1699 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1417762258432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31980668 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98193723380736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31980668 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191746048 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98193705377792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95631116318720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1488962415104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1417105883648 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98193705377792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56510260 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99611485639168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884295828593855e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22439229 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108763693703680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107274731288576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97043678179315 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22439674 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98193723380736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99610811261440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24529592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14309220 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24529590 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24529592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167805952 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56490196 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1412544821235 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14309222 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚1699 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1417762258432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31980668 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98193723380736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31980668 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191746048 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98193705377792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95631116318720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1488962415104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1417105883648 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98193705377792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56510260 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99611485639168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884295828593855e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56510260 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0‰~(±2»- _wandbruntime‚1267  _runtime‚1267A׺S¼Í /loss/FRt3/assignment_loss‚5.182729850496565 2loss/SRqqt2/assignment_loss‚1.7714287987818904 .loss/FRt1/detection_loss‚1.720112268282817 1loss/SRqqt1/detection_loss‚1.2053527816771024 /loss/FBt4/detection_loss‚2.1934284307062626 /loss/FRt1/assignment_loss‚2.681760023190425 /loss/FRt4/assignment_loss‚3.873338268353389 1loss/SRqqt4/assignment_loss‚2.284536687142037 0loss/FBt1/assignment_loss‚0.3036283953183124 0loss/FBt3/assignment_loss‚0.5621149738629643 2loss/SRqqt1/assignment_loss‚1.4884186939347581 .loss/FBt3/detection_loss‚3.435205360253652 trainer/global_step‚1699 .loss/FRt4/detection_loss‚2.651295768144803 1loss/SRqqt2/detection_loss‚1.8951049593149447 0loss/SRqqt3/detection_loss‚2.428983211517334 %loss/total_loss‚53.86213621992508 /loss/FRt2/assignment_loss‚4.222637077172597 0loss/FBt2/assignment_loss‚0.6044689335879715 2loss/SRqqt3/assignment_loss‚2.6283020871026177 0loss/SRqqt4/detection_loss‚2.017443721281721 .loss/FBt1/detection_loss‚0.728810437784983 /loss/FBt2/detection_loss‚3.9849691887696586 epoch‚1 0loss/FBt4/assignment_loss‚0.8762378133833302 .loss/FRt3/detection_loss‚2.870030801636832 % _timestamp‚1.7884295828611445e+09 _step‚138 /loss/FRt2/detection_loss‚2.2517976882281125 _runtime‚1267.565791972þòúX½Ò 2loss/SRqqt1/assignment_loss‚1.4884186939347581 .loss/FBt3/detection_loss‚3.435205360253652 trainer/global_step‚1699 .loss/FRt4/detection_loss‚2.651295768144803 1loss/SRqqt2/detection_loss‚1.8951049593149447 0loss/SRqqt3/detection_loss‚2.428983211517334 %loss/total_loss‚53.86213621992508 /loss/FRt2/assignment_loss‚4.222637077172597 0loss/FBt2/assignment_loss‚0.6044689335879715 2loss/SRqqt3/assignment_loss‚2.6283020871026177 0loss/SRqqt4/detection_loss‚2.017443721281721 .loss/FBt1/detection_loss‚0.728810437784983 /loss/FBt2/detection_loss‚3.9849691887696586 epoch‚1 0loss/FBt4/assignment_loss‚0.8762378133833302 .loss/FRt3/detection_loss‚2.870030801636832 % _timestamp‚1.7884295828611445e+09 _step‚138 /loss/FRt2/detection_loss‚2.2517976882281125 _runtime‚1267.565791972 /loss/FRt3/assignment_loss‚5.182729850496565 2loss/SRqqt2/assignment_loss‚1.7714287987818904 .loss/FRt1/detection_loss‚1.720112268282817 1loss/SRqqt1/detection_loss‚1.2053527816771024 /loss/FBt4/detection_loss‚2.1934284307062626 /loss/FRt1/assignment_loss‚2.681760023190425 /loss/FRt4/assignment_loss‚3.873338268353389 1loss/SRqqt4/assignment_loss‚2.284536687142037 0loss/FBt1/assignment_loss‚0.3036283953183124 0loss/FBt3/assignment_loss‚0.5621149738629643Šµˆ®Y“¾‚(:ˆ –ŠåԘ떎 gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚101.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò'¡"¿‚(:— –ŠåÔ¡¶‡‘ disk.drbd1777.out‚0% disk./.usageGB‚1099.7203369140625 disk.rbd0.in‚0 disk.dm-0.out‚6672.732421875 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 disk.rbd4.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd2.out‚61.11767578125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda3.out‚ 53.05078125 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 53.05078125$ memory_percent‚99.95555877685547 disk.sda1.in‚0 network.sent‚ 2.706502e+06 network.recv‚516413 disk.dm-0.in‚0 disk.sda.out‚ 53.05078125 cpu‚25.273750243507322 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda.in‚0# disk.nvme0n1.out‚6672.732421875 disk.rbd1.out‚ 0.61328125# proc.memory.rssMB‚ 2862.51953125) proc.memory.percent‚17.47143268585205) disk./.usagePercent‚73.80315453712527 disk.sda3.in‚0 disk.nvme0n1.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.rbd1.in‚0% disk.nvme0c0n1.out‚6672.732421875# proc.memory.availableMB‚7.28125„öOú“À‚(:ˆ ¥ŠåÔì¿ó gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.568' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.5136 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬mŸÏ$Á‚(:™ ¥ŠåԼĖ) disk./.usagePercent‚73.80315607317047 disk.dm-0.out‚6771.779296875 disk.rbd2.in‚0$ memory_percent‚99.95720386505127 disk.dm-1.in‚0 disk.dm-1.out‚ 53.66015625 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0) proc.memory.percent‚17.47143268585205$ disk./.usageGB‚1099.720359802246 disk.dm-0.in‚0 disk.sda.in‚0 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda.out‚ 53.66015625 disk.rbd1.out‚ 0.61328125 disk.sda3.out‚ 53.66015625 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2862.51953125 network.sent‚ 2.815126e+06 network.recv‚523117 disk.rbd3.out‚0 disk.nvme0n1.in‚0# disk.nvme0n1.out‚6771.779296875% disk.nvme0c0n1.out‚6771.779296875 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 cpu‚25.271044728361755 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd2.out‚62.62939453125& proc.memory.availableMB‚ 7.01171875 disk.rbd0.out‚0¥Ã'2Â- _wandbruntime‚1302  _runtime‚1302ó YÉÃà ' lr-AdamW/pg1‚0.00022524840606449423 ' lr-AdamW/pg2‚0.00022524840606449423 trainer/global_step‚1749 % _timestamp‚1.7884296174755673e+09 _runtime‚1302.180275347 _step‚139¿?^`ÎÄÈ trainer/global_step‚1749 % _timestamp‚1.7884296174755673e+09 _runtime‚1302.180275347 _step‚139 ' lr-AdamW/pg1‚0.00022524840606449423 ' lr-AdamW/pg2‚0.00022524840606449423‹^M2Å- _wandbruntime‚1302  _runtime‚1302Ì?yûÓ'ÆÍO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102412950907392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98321435755520 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25216374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1457272261632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58104616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25216376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23062576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚1302.181275993 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102412276542976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1457928623104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98321418716160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1457928623104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚1749 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100955004281344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1451909287723 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1531079217152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100955022284288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25216374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32888240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100955022284288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1452563725471 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32868204 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1531238647808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14709454 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111826856081920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8353554 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25216376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110295617434112 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚99773999480991 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58084578 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102412276542976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1457928623104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98321418716160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1457928623104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚1749 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100955004281344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1451909287723 JBDeviceStatsMonitor.on_train_batch_start/segment.×øžÏsmall_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1531079217152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100955022284288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25216374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32888240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100955022284288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1452563725471 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32868204 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1531238647808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14709454 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111826856081920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8353554ŒÜök2È- _wandbruntime‚1302  _runtime‚1302Û>Ú;Ý&É×M M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32886658 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110357268171776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58137112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102469449009664 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚99829720046723 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58137112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1458748064256 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚102470123387392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25230392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14717620 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102470123387392 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1532100302336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25230392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1531922010624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32886658 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1452725722687 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25230390 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1458091689472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98376339886080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚99829048569407 % _timestamp‚1.7884296181364813e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32906720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23075650 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110357244231680 _step‚141 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101011375323136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1458091689472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101011357320192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58117048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14717622 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23075201 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚98376322846720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8358028 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58117048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101011375323136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111889166242304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1453380160643 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32906720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1458748064256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101011357320192Ÿ/¹¼â&ÊÜM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25230390 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8357581 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1302.841891777 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58137112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102469449009664 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚99829720046723 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58137112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1458748064256 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚102470123387392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25230392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14717620 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102470123387392 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1532100302336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25230392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1531922010624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32886658 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1452725722687 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25230390 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1458091689472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98376339886080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚99829048569407 % _timestamp‚1.7884296181364813e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32906720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23075650 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110357244231680 _step‚141 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101011375323136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1458091689472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101011357320192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58117048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14717622 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23075201 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚98376322846720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8358028 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58117048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101011375323136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111889166242304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1453380160643 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32906720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1458748064256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101011357320192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32886658 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110357268171776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 trainer/global_step‚1799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25917174 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23699887 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114952387907584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105270914291200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105271588655616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1574233590272 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113378130377216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105270914291200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33814292 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103772656223744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25917174 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114952202439680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33794256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103772674226688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101066625244160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚101066642283520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚1337.305688445 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15117854 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8581615 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113378154317312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59731468 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23699467 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25917176 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33794256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15117852 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1498258067456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527808³ºÛ'ÙÕO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1498258067456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 trainer/global_step‚1799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25917174 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23699887 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114952387907584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105270914291200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105271588655616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1574233590272 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113378130377216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105270914291200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33814292 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103772656223744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25917174 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114952202439680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33794256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103772674226688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101066625244160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚101066642283520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚1337.305688445 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15117854 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8581615 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113378154317312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59731468 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23699467 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25917176 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33794256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15117852 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928v‡¬=2Ú- _wandbruntime‚1337  _runtime‚1337Â*ˆgá&ÛÛM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102615090436815 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15126022 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101121529374720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105328761135616 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33812710 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1499077495296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103829027265536 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25931190 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1499077495296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115014703057920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1494215500051 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105328086757888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1574921943040 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25931192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23712111 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚145 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103829027265536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25931190 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101121546414080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59763964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115014898998272 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884296532527301e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33832772 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1499077495296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103829027265536 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25931190 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1499077495296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115014703057920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1494215500051 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105328086757888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1574921943040 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25931192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23712111 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚145 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103829027265536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25931190 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101121546414080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59763964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115014898998272 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908‘FÆ'2Ý- _wandbruntime‚1337  _runtime‚1337¨wìØYÞÓ 0loss/FRt2/assignment_loss‚3.8783199251914513 2loss/SRqqt4/assignment_loss‚2.4609694734532783 1loss/SRqqt4/detection_loss‚2.0832048128259943 0loss/FRt1/assignment_loss‚2.2684939905088775 %loss/total_loss‚55.34830158204565 % _timestamp‚1.7884296532546098e+09 _runtime‚1337.959235726 .loss/FRt2/detection_loss‚2.275800251839112 2loss/SRqqt1/assignment_loss‚1.4680865890404675 2loss/SRqqt2/assignment_loss‚2.1406253383805356 0loss/FBt1/assignment_loss‚0.2336910086579756 .loss/FRt1/detection_loss‚2.119926538698527 0loss/SRqqt3/detection_loss‚2.661136109095353 /loss/FBt1/detection_loss‚0.9304626152017359 0loss/SRqqt3/assignment_loss‚2.38776788688623 0loss/FBt4/assignment_loss‚0.6929281051917415 /loss/FRt3/detection_loss‚3.2567484864482172 /loss/FRt4/detection_loss‚2.5977556024278914 epoch‚1 2loss/FBt2/assignment_loss‚0.010423092171549797 1loss/SRqqt1/detection_loss‚1.5465665446827188 _step‚146 -loss/FBt4/detection_loss‚2.43587689101696 /loss/FRt3/assignment_loss‚4.960520170353077 0loss/FRt4/assignment_loss‚3.9656523636409213 1loss/SRqqt2/detection_loss‚1.9016202986240387 /loss/FBt3/detection_loss‚2.8533616799574633 0loss/FBt3/assignment_loss‚0.5532237749832797 /loss/FBt2/detection_loss‚5.6651400327682495 trainer/global_step‚1799„ ëÁ^ßØ % _timestamp‚1.7884296532546098e+09 _runtime‚1337.959235726 .loss/FRt2/detection_loss‚2.275800251839112 2loss/SRqqt1/assignment_loss‚1.4680865890404675 2loss/SRqqt2/assignment_loss‚2.1406253383805356 0loss/FBt1/assignment_loss‚0.2336910086579756 .loss/FRt1/detection_loss‚2.119926538698527 0loss/SRqqt3/detection_loss‚2.661136109095353 /loss/FBt1/detection_loss‚0.9304626152017359 0loss/SRqqt3/assignment_loss‚2.38776788688623 0loss/FBt4/assignment_loss‚0.6929281051917415 /loss/FRt3/detection_loss‚3.2567484864482172 /loss/FRt4/detection_loss‚2.5977556024278914 epoch‚1 2loss/FBt2/assignment_loss‚0.010423092171549797 1loss/SRqqt1/detection_loss‚1.5465665446827188 _step‚146 -loss/FBt4/detection_loss‚2.43587689101696 /loss/FRt3/assignment_loss‚4.960520170353077 0loss/FRt4/assignment_loss‚3.9656523636409213 1loss/SRqqt2/detection_loss‚1.9016202986240387 /loss/FBt3/detection_loss‚2.8533616799574633 0loss/FBt3/assignment_loss‚0.5532237749832797 /loss/FBt2/detection_loss‚5.6651400327682495 trainer/global_step‚1799 0loss/FRt2/assignment_loss‚3.8783199251914513 2loss/SRqqt4/assignment_loss‚2.4609694734532783 1loss/SRqqt4/detection_loss‚2.0832048128259943 0loss/FRt1/assignment_loss‚2.2684939905088775 %loss/total_loss‚55.34830158204565’T'¯üà‚(:’ áŠåÔü§€Ž gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.681' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.536199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë„rÂ%á‚(:š áŠåÔµºû% disk./.usageGB‚1099.7193603515625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.94064e+06 disk.nvme0n1.in‚0' disk.nvme0c0n1.out‚7071.39794921875 disk.sda3.in‚0 disk.rbd4.out‚0 disk.sda.out‚ 56.2109375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd0.in‚0$ memory_percent‚99.95758533477783 disk.rbd2.in‚0 disk.drbd1777.out‚0 cpu‚25.261194352318192 network.recv‚541828 disk.rbd0.out‚0 disk.rbd2.out‚ 62.9970703125 disk.rbd3.out‚0" disk.dm-0.out‚7071.39794921875% disk.nvme0n1.out‚7071.39794921875 disk.sda3.out‚ 56.2109375 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.76171875 disk.dm-0.in‚0 disk.sda.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 6.94921875# proc.memory.rssMB‚ 2862.51953125) proc.memory.percent‚17.47143268585205( disk./.usagePercent‚73.8030889991965 disk.dm-1.out‚ 56.2109375 disk.sda2.in‚0 disk.drbd1777.in‚0@àÿ ’â‚(:‡ ðŠåԧфŽ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚89.787' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßüÌ #ã‚(:˜ ðŠåÔÚë‘ disk.drbd1777.out‚0# disk.nvme0n1.out‚7072.048828125 disk.rbd1.out‚ 0.76171875$ memory_percent‚99.95646476745605 cpu‚25.258884109147505 network.sent‚ 3.048502e+06) disk./.usagePercent‚73.80072348958012 disk.rbd3.in‚0% proc.memory.availableMB‚ 7.1328125) proc.memory.percent‚17.47143268585205 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda2.in‚0 disk.dm-0.out‚7072.048828125 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0% disk./.usageGB‚1099.6841125488281 disk.sda3.out‚ 56.71484375 disk.sda1.out‚0 disk.rbd2.out‚ 64.6494140625 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.out‚ 56.71484375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2862.51953125 network.recv‚547952 disk.rbd4.in‚0 disk.dm-1.out‚ 56.71484375 disk.nvme0n1.in‚0 disk.dm-0.in‚0% disk.nvme0c0n1.out‚7072.048828125iƒ½V2ä- _wandbruntime‚1372  _runtime‚1372dópÈå ' lr-AdamW/pg1‚0.00022469381834552303 ' lr-AdamW/pg2‚0.00022469381834552303 trainer/global_step‚1849 $ _timestamp‚1.788429688043035e+09 _runtime‚1372.747802303 _step‚147ò¼èþÍæÇ ' lr-AdamW/pg1‚0.00022469381834552303 ' lr-AdamW/pg2‚0.00022469381834552303 trainer/global_step‚1849 $ _timestamp‚1.788429688043035e+09 _runtime‚1372.747802303 _step‚147“ô$2ç- _wandbruntime‚1372  _runtime‚1372ˆ„‹Ê×'èÑO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24336289 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116460691200512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚106590308166144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1539900234752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34720308 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1617179271168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103811848811520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103811831772160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26617974 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108129552039424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8810499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34740344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1534234404287 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 trainer/global_step‚1849 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108130226403840 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24336753 % _timestamp‚1.7884296880438535e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26617976 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15526254 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108130226403840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118077846531584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34740344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34720308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15526252 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1539243873280 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61338282 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1539900234752 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61358320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8810037 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105345411738699 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26617974 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚106590326169088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 _runtime‚1372.748747524 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195953664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116460667260416 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24336289 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116460691200512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚106590308166144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1539900234752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34720308 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1617179271168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103811848811520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103811831772160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26617974 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108129552039424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8810499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34740344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1534234404287 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 trainer/global_step‚1849 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108130226403840 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24336753 % _timestamp‚1.7884296880438535e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26617976 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15526254 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108130226403840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118077846531584 WJDeviceStͱƒatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34740344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34720308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15526252 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1539243873280 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61338282 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1539900234752 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61358320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0”O p2ê- _wandbruntime‚1373  _runtime‚1373§þá&ëÛM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34758824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105401132304223 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108187398883840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8815047 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚118140549621248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103866752942080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61370752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15534422 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1535050839459 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1540719675904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1849 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚106646679207936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34738762 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540063301120 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24349019 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108186724506112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24349469 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1618207683072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61390816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1618044071424ʧ,æ&ìàM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34758824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105401132304223 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108187398883840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8815047 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚118140549621248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103866752942080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61370752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15534422 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1535050839459 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1540719675904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1849 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚106646679207936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34738762 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540063301120 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24349019 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108186724506112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24349469 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1618207683072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61390816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1618044071424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414•´Êbj2í- _wandbruntime‚1373  _runtime‚1373ó•YîÓ 0loss/FRt3/assignment_loss‚4.5231655325208395 2loss/SRqqt1/assignment_loss‚1.4722885846355815 2loss/SRqqt4/assignment_loss‚2.0066308640703863 /loss/FRt4/detection_loss‚2.3904038701545107 1loss/SRqqt1/detection_loss‚1.2784474433204274 0loss/SRqqt3/detection_loss‚2.367412683126089 /loss/FBt1/detection_loss‚0.6867871449756201 .loss/FBt3/detection_loss‚4.457882714271546 % _timestamp‚1.7884296887041652e+09 2loss/SRqqt3/assignment_loss‚2.5945548028559298 0loss/FBt2/assignment_loss‚0.6487664286978543 _step‚150 2loss/SRqqt2/assignment_loss‚1.9962023059104352 1loss/SRqqt4/detection_loss‚2.0965155399575526 trainer/global_step‚1849 /loss/FRt1/assignment_loss‚2.571518851261513 0loss/FBt4/assignment_loss‚0.4124787338078022 /loss/FRt1/detection_loss‚1.9595053104647235 .loss/FRt2/detection_loss‚2.536800939412344 /loss/FBt2/detection_loss‚3.3451228936513266 %loss/total_loss‚53.41278558091743 _runtime‚1373.408775359 0loss/FRt2/assignment_loss‚3.6551803946495056 /loss/FRt4/assignment_loss‚3.963011300021952 0loss/FBt3/assignment_loss‚0.5282630503177629 /loss/FBt4/detection_loss‚2.3112074956297874 epoch‚1 0loss/FBt1/assignment_loss‚0.2344692682348304 .loss/FRt3/detection_loss‚3.430401066939036 1loss/SRqqt2/detection_loss‚1.9457683620300699Ûl^ïØ /loss/FRt4/assignment_loss‚3.963011300021952 0loss/FBt3/assignment_loss‚0.5282630503177629 /loss/FBt4/detection_loss‚2.3112074956297874 epoch‚1 0loss/FBt1/assignment_loss‚0.2344692682348304 .loss/FRt3/detection_loss‚3.430401066939036 1loss/SRqqt2/detection_loss‚1.9457683620300699 0loss/FRt3/assignment_loss‚4.5231655325208395 2loss/SRqqt1/assignment_loss‚1.4722885846355815 2loss/SRqqt4/assignment_loss‚2.0066308640703863 /loss/FRt4/detection_loss‚2.3904038701545107 1loss/SRqqt1/detection_loss‚1.2784474433204274 0loss/SRqqt3/detection_loss‚2.367412683126089 /loss/FBt1/detection_loss‚0.6867871449756201 .loss/FBt3/detection_loss‚4.457882714271546 % _timestamp‚1.7884296887041652e+09 2loss/SRqqt3/assignment_loss‚2.5945548028559298 0loss/FBt2/assignment_loss‚0.6487664286978543 _step‚150 2loss/SRqqt2/assignment_loss‚1.9962023059104352 1loss/SRqqt4/detection_loss‚2.0965155399575526 trainer/global_step‚1849 /loss/FRt1/assignment_loss‚2.571518851261513 0loss/FBt4/assignment_loss‚0.4124787338078022 /loss/FRt1/detection_loss‚1.9595053104647235 .loss/FRt2/detection_loss‚2.536800939412344 /loss/FBt2/detection_loss‚3.3451228936513266 %loss/total_loss‚53.41278558091743 _runtime‚1373.408775359 0loss/FRt2/assignment_loss‚3.6551803946495056–3$sÞð‚(:“ ÿŠåÔõ÷é gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.002' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.800400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OE5$ñ‚(:™ ÿŠåÔàÈÑ disk.sda.out‚ 57.44921875 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0) disk./.usagePercent‚73.80071964946711 disk.rbd1.out‚ 0.7939453125 disk.rbd0.in‚0 disk.sda3.in‚0 disk.dm-0.in‚0 disk.sda1.in‚0# disk.nvme0n1.out‚7171.142578125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 57.44921875 proc.cpu.threads‚17 disk.sda3.out‚ 57.44921875 disk.dm-1.in‚0$ memory_percent‚99.95808601379395% proc.memory.availableMB‚ 6.8671875# proc.memory.rssMB‚ 2862.51953125 disk.dm-0.out‚7171.142578125 network.recv‚552262% disk./.usageGB‚1099.6840553283691 disk.rbd2.out‚ 64.6494140625 disk.drbd1777.out‚0) proc.memory.percent‚17.47143268585205 cpu‚25.25670039185903 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd4.out‚0% disk.nvme0c0n1.out‚7171.142578125 disk.drbd1777.in‚0 disk.rbd4.in‚0 network.sent‚ 3.054733e+06 disk.rbd3.out‚0 disk.sda.in‚0$%¿P’ò‚(:‡ Ž‹åÔæÛÝ gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥&Š"ó‚(:— Ž‹åÔ¯¤ú) proc.memory.percent‚17.47143268585205 disk.sda3.in‚0 disk.rbd4.out‚0 disk.sda.out‚57.90625$ memory_percent‚99.95665550231934 network.sent‚ 3.162379e+06 disk.sda3.out‚57.90625 disk.sda1.in‚0" disk.dm-0.out‚7270.74267578125 disk.dm-1.in‚0% disk.nvme0n1.out‚7270.74267578125# proc.memory.rssMB‚ 2862.51953125 cpu‚25.254507008774894 network.recv‚558048 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚7270.74267578125 disk.sda2.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1099.6842346191406 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.sda2.in‚0) disk./.usagePercent‚73.80073168182122 disk.rbd2.in‚0 disk.rbd2.out‚ 64.7275390625 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.dm-1.out‚57.90625 disk.rbd1.out‚ 0.7939453125 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.dm-0.in‚0% proc.memory.availableMB‚ 7.1015625n.ò?2ô- _wandbruntime‚1408  _runtime‚1408Ïüy6ÇõÁ $ _timestamp‚1.788429723815121e+09 _runtime‚ 1408.51990435 _step‚151 ' lr-AdamW/pg1‚0.00022438167648724595 ' lr-AdamW/pg2‚0.00022438167648724595 trainer/global_step‚1899œI ¹ÌöÆ ' lr-AdamW/pg1‚0.00022438167648724595 ' lr-AdamW/pg2‚0.00022438167648724595 trainer/global_step‚1899 $ _timestamp‚1.788429723815121e+09 _runtime‚ 1408.51990435 _step‚151—Yku2÷- _wandbruntime‚1408  _runtime‚1408*r7†±øÐO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27318774 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109407978111488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27318776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121203431720448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109407960108544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106557055339520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121203627674112 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109407960108544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24974102 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35646360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119543228083712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119543204143616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62965134 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1580886040576 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚110988189787648 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1580229679104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884297238158722e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_po©q‡–%ol.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚110988864152064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195953664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106557038300160 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1660227576832 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1899 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110988189787648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9039008 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108131453606107 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109407978111488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1574415305947 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110988189787648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9039008 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108131453606107 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109407978111488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1574415305947 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109407960108544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106557055339520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121203627674112 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109407960108544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24974102 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414˜H¦G2ú- _wandbruntime‚1409  _runtime‚1409 eÿØá&ûÛM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9044029 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108187174171631 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111045362254336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1581049106944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121265933305344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109464331150336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _step‚153 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9043569 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1575231740911 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24986389 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35684876 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119604854881280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884297245018005e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63017668 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27332792 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27332790 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111046036632064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111046036632064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109464331150336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169903104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35684876 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63017668 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121266127148544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1581705481728 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62997604 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35664814 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193843200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1575886178867 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚1409.207083853 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109464313147392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27332792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119604878821376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35664814 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24986851 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ƒC7ÊüàM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9043569 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1575231740911 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24986389 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35684876 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119604854881280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884297245018005e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63017668 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27332792 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27332790 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111046036632064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111046036632064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109464331150336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169903104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35684876 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63017668 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121266127148544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1581705481728 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62997604 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35664814 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193843200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1575886178867 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚1409.207083853 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109464313147392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27332792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 GADeviceStatsMonitor.on_train_batch_end/oversize_segments íü .allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119604878821376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35664814 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24986851 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9044029 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108187174171631 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111045362254336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1581049106944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121265933305344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109464331150336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _step‚153™³g-2ý- _wandbruntime‚1409  _runtime‚1409=´TþÎ /loss/FRt1/assignment_loss‚2.387604296207428 2loss/SRqqt1/assignment_loss‚1.4312917763410613 2loss/SRqqt2/assignment_loss‚2.1524929881972423 /loss/FRt1/detection_loss‚1.7472446421161294 2loss/SRqqt4/assignment_loss‚1.7413463830807299 $ _timestamp‚1.788429724503565e+09 /loss/FRt2/assignment_loss‚4.472315893173218 0loss/FBt2/assignment_loss‚0.7337658355633417 /loss/FRt2/detection_loss‚2.3290443390607836 %loss/total_loss‚58.11812033530339 _step‚154 /loss/FBt3/detection_loss‚2.9306261972947554 trainer/global_step‚1899 1loss/SRqqt3/assignment_loss‚2.459063040799108 0loss/FBt1/assignment_loss‚0.3343553867938924 0loss/FBt4/assignment_loss‚0.5187429040670395 1loss/SRqqt1/detection_loss‚1.1340935959718947 epoch‚1 /loss/FRt4/assignment_loss‚3.960859501684034 0loss/FBt3/assignment_loss‚0.7148676622997475 /loss/FRt4/detection_loss‚2.6599763865406447 /loss/FBt2/detection_loss‚3.4659536282221475 _runtime‚1409.208502486 /loss/FRt3/assignment_loss‚5.342051351070404 .loss/FRt3/detection_loss‚3.429819294810295 0loss/SRqqt4/detection_loss‚1.930032546790141 /loss/FBt1/detection_loss‚1.1421599784737015 -loss/FBt4/detection_loss‚6.22939632833004 1loss/SRqqt2/detection_loss‚1.9099986074017543 0loss/SRqqt3/detection_loss‚2.961017771013852íb\ÛYÿÓ /loss/FRt4/assignment_loss‚3.960859501684034 0loss/FBt3/assignment_loss‚0.7148676622997475 /loss/FRt4/detection_loss‚2.6599763865406447 /loss/FBt2/detection_loss‚3.4659536282221475 _runtime‚1409.208502486 /loss/FRt3/assignment_loss‚5.342051351070404 .loss/FRt3/detection_loss‚3.429819294810295 0loss/SRqqt4/detection_loss‚1.930032546790141 /loss/FBt1/detection_loss‚1.1421599784737015 -loss/FBt4/detection_loss‚6.22939632833004 1loss/SRqqt2/detection_loss‚1.9099986074017543 0loss/SRqqt3/detection_loss‚2.961017771013852 /loss/FRt1/assignment_loss‚2.387604296207428 2loss/SRqqt1/assignment_loss‚1.4312917763410613 2loss/SRqqt2/assignment_loss‚2.1524929881972423 /loss/FRt1/detection_loss‚1.7472446421161294 2loss/SRqqt4/assignment_loss‚1.7413463830807299 $ _timestamp‚1.788429724503565e+09 /loss/FRt2/assignment_loss‚4.472315893173218 0loss/FBt2/assignment_loss‚0.7337658355633417 /loss/FRt2/detection_loss‚2.3290443390607836 %loss/total_loss‚58.11812033530339 _step‚154 /loss/FBt3/detection_loss‚2.9306261972947554 trainer/global_step‚1899 1loss/SRqqt3/assignment_loss‚2.459063040799108 0loss/FBt1/assignment_loss‚0.3343553867938924 0loss/FBt4/assignment_loss‚0.5187429040670395 1loss/SRqqt1/detection_loss‚1.1340935959718947 epoch‚1šø#pÞ€‚(:“ ‹åÔŸ÷ñ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚128.027' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.605399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼|Á[*‚(:Ÿ ‹åÔë»´‘ disk.dm-0.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd4.in‚0) proc.memory.percent‚17.47143268585205 cpu‚25.25235350968046% disk./.usageGB‚1099.6841773986816" disk.dm-0.out‚7369.79345703125 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 network.recv‚563234 disk.sda.out‚ 58.83203125 disk.dm-1.out‚ 58.83203125 disk.drbd1777.out‚0# memory_percent‚99.9565601348877% proc.memory.availableMB‚ 7.1171875 network.sent‚ 3.169084e+06 disk.sda1.in‚0' disk.nvme0c0n1.out‚7369.79345703125 disk.rbd2.in‚0) disk./.usagePercent‚73.80072784170821 disk.sda.in‚0 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.85693359375 disk.rbd3.out‚0 disk.rbd2.out‚ 64.7275390625 disk.sda3.out‚ 58.83203125 disk.drbd1777.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚7369.79345703125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2862.51953125 proc.cpu.threads‚17 disk.rbd4.out‚0ÃÚ@ž‚‚(:“ ¬‹åÔ¬Ú„Ž gpu.0.gpu‚33.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.762' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.152400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v®t"ƒ‚(:— ¬‹åÔ¨¨» disk.sda.out‚59.59375 disk.dm-0.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0 disk.sda3.out‚59.59375) proc.memory.percent‚17.47143268585205 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚64.82568359375$ memory_percent‚99.95861053466797' disk.nvme0c0n1.out‚7370.23779296875 disk.rbd4.in‚0 disk.sda1.out‚0 disk.dm-1.out‚59.59375% disk.nvme0n1.out‚7370.23779296875# proc.memory.rssMB‚ 2862.51953125 disk.nvme0c0n1.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0% disk./.usageGB‚1099.6841773986816 disk.rbd4.out‚0" disk.dm-0.out‚7370.23779296875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 network.recv‚568956) disk./.usagePercent‚73.80072784170821 disk.rbd3.out‚0 disk.sda1.in‚0# proc.memory.availableMB‚6.78125 cpu‚25.250296037116172 proc.cpu.threads‚17 network.sent‚ 3.276756e+06 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.85693359375FX„‚(:’ »‹åÔšŠè gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.045999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ’+…‚(:  »‹åÔ“ºÃ disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 0.919921875$ memory_percent‚99.95865821838379 disk.sda3.out‚ 60.01171875 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0% disk.nvme0n1.out‚7469.31201171875 disk.sda2.out‚0% disk./.usageGB‚1099.6842193603516 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.sda.out‚ 60.01171875 disk.nvme0c0n1.in‚0) proc.memory.percent‚17.47143268585205 proc.cpu.threads‚17 network.sent‚ 3.283054e+06 disk.rbd4.out‚0 disk.sda2.in‚0 network.recv‚573339) disk./.usagePercent‚73.80073065779108 disk.sda3.in‚0 disk.dm-0.in‚0# proc.memory.rssMB‚ 2862.51953125 disk.dm-1.out‚ 60.01171875 disk.rbd2.out‚64.82568359375% proc.memory.availableMB‚ 6.7734375" disk.dm-0.out‚7469.31201171875 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚7469.31201171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 cpu‚25.248244464249186[3J2†- _wandbruntime‚1444  _runtime‚1444—µÅøLJÁ & lr-AdamW/pg1‚0.0002240463877505247 & lr-AdamW/pg2‚0.0002240463877505247 trainer/global_step‚1949 % _timestamp‚1.7884297598328989e+09 _runtime‚1444.537700373 _step‚155ÕTÌcÌˆÆ & lr-AdamW/pg1‚0.0002240463877505247 & lr-AdamW/pg2‚0.0002240463877505247 trainer/global_step‚1949 % _timestamp‚1.7884297598328989e+09 _runtime‚1444.537700373 _step‚155›¾ŸÇ­2‰- _wandbruntime‚1444  _runtime‚1444.¼_é×'ŠÑO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1621215484928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25611498 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9268446 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1703456842752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113847501900288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112225630053888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36572412 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚156 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36592448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109302261867520 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113846827535872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112225630053888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16343054 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1621871846400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122625764966912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 _runtime‚1444.538667596 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28019574 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64591986 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113847501900288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122625741026816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124329030050304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1703289023488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113846827535872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64612024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9268911 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 trainer/global_step‚1949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112225612050944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1621871846400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36572412 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚156 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36592448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109302261867520 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113846827535872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112225630053888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16343054 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1621871846400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122625764966912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 _runtime‚1444.538667596 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28019574 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64591986 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113847501900288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122625741026816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124329030050304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1703289023488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113846827535872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64612024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9268911 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 trainer/global_step‚1949œò `2Œ- _wandbruntime‚1445  _runtime‚1445GéѶ[ÚM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113904674380288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1622691287552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28033592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179163136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64624456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122687391764480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9273046 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_||Ï{…!train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36610928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36610928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1616067080319 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1622034912768 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112281983092736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122687415704576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113904674380288 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28033590 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36590866 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16351220 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110973216039039 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112281983092736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110973887516355 _step‚157 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112281965089792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1704323274240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25624266 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16351222 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112281965089792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109357148958720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28033590 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9273510 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109357165998080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1616721518275 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113904000002560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1622691287552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155223040 trainer/global_step‚1949 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64644520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872Ý…j‹å&ŽßM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113904000002560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1622691287552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155223040 trainer/global_step‚1949 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64644520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113904674380288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1622691287552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28033592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179163136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64624456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122687391764480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9273046 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36610928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36610928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1616067080319 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1622034912768 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112281983092736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122687415704576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113904674380288 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28033590 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36590866 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16351220 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110973216039039 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112281983092736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110973887516355 _step‚157 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112281965089792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1704323274240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25624266 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16351222 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112281965089792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109357148958720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28033590 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9273510 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109357165998080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1616721518275 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513†z•*2- _wandbruntime‚1445  _runtime‚1445…æÅ|RÌ 1loss/SRqqt3/detection_loss‚2.5687489961755685 /loss/FRt2/assignment_loss‚4.038098935987435 1loss/SRqqt1/assignment_loss‚1.252255428949712 1loss/SRqqt1/detection_loss‚1.2049515661849102 0loss/FRt1/assignment_loss‚3.0011494479797505 /loss/FRt4/assignment_loss‚4.657484447956085 2loss/SRqqt3/assignment_loss‚2.5875580434141487 .loss/FRt2/detection_loss‚2.308035312914381 .loss/FRt4/detection_loss‚2.252383671998978 .loss/FBt3/detection_loss‚3.765433589617411 .loss/FBt4/detection_loss‚4.008606817041125 0loss/SRqqt2/detection_loss‚2.143224561586976 /loss/FBt1/detection_loss‚1.1690368032827514 .loss/FBt2/detection_loss‚4.116763075192769 % _timestamp‚1.7884297604936144e+09 1loss/SRqqt2/assignment_loss‚2.165985411653916 2loss/SRqqt4/assignment_loss‚1.7510421773926779 0loss/FBt1/assignment_loss‚0.2541841480072298 .loss/FRt1/detection_loss‚2.031150516260553 %loss/total_loss‚57.49870432378502 trainer/global_step‚1949 _runtime‚1445.198254489 _step‚158 0loss/FBt4/assignment_loss‚0.3569543489388047 .loss/FRt3/detection_loss‚3.199098961353302 1loss/SRqqt4/detection_loss‚1.9864026667042212 epoch‚1 /loss/FRt3/assignment_loss‚5.667954549789429 1loss/FBt2/assignment_loss‚0.33127312175929546 /loss/FBt3/assignment_loss‚0.680927723643587©H7W‘Ñ /loss/FRt4/assignment_loss‚4.657484447956085 2loss/SRqqt3/assignment_loss‚2.5875580434141487 .loss/FRt2/detection_loss‚2.308035312914381 .loss/FRt4/detection_loss‚2.252383671998978 .loss/FBt3/detection_loss‚3.765433589617411 .loss/FBt4/detection_loss‚4.008606817041125 0loss/SRqqt2/detection_loss‚2.143224561586976 /loss/FBt1/detection_loss‚1.1690368032827514 .loss/FBt2/detection_loss‚4.116763075192769 % _timestamp‚1.7884297604936144e+09 1loss/SRqqt2/assignment_loss‚2.165985411653916 2loss/SRqqt4/assignment_loss‚1.7510421773926779 0loss/FBt1/assignment_loss‚0.2541841480072298 .loss/FRt1/detection_loss‚2.031150516260553 %loss/total_loss‚57.49870432378502 trainer/global_step‚1949 _runtime‚1445.198254489 _step‚158 0loss/FBt4/assignment_loss‚0.3569543489388047 .loss/FRt3/detection_loss‚3.199098961353302 1loss/SRqqt4/detection_loss‚1.9864026667042212 epoch‚1 /loss/FRt3/assignment_loss‚5.667954549789429 1loss/FBt2/assignment_loss‚0.33127312175929546 /loss/FBt3/assignment_loss‚0.680927723643587 1loss/SRqqt3/detection_loss‚2.5687489961755685 /loss/FRt2/assignment_loss‚4.038098935987435 1loss/SRqqt1/assignment_loss‚1.252255428949712 1loss/SRqqt1/detection_loss‚1.2049515661849102 0loss/FRt1/assignment_loss‚3.0011494479797505žµËzž’‚(:“ Ê‹åÔɳˎ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.158' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.831599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01\ $“‚(:™ Ê‹åÔ®ªí network.recv‚576873) disk./.usagePercent‚73.80070864114315 disk.sda.out‚ 60.390625) proc.memory.percent‚17.47143268585205 cpu‚25.24622531931234 disk.sda1.in‚0% disk.nvme0n1.out‚7569.25830078125 disk.rbd2.in‚0 disk.rbd2.out‚66.41552734375 network.sent‚ 3.28838e+06 disk.rbd4.in‚0" disk.dm-0.out‚7569.25830078125 disk.rbd1.out‚ 0.919921875 disk.dm-0.in‚0 disk.dm-1.out‚ 60.390625 disk.rbd1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0& proc.memory.availableMB‚ 6.77734375 disk.sda.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚7569.25830078125 disk.sda2.out‚0 disk.sda3.out‚ 60.390625# proc.memory.rssMB‚ 2862.51953125 proc.cpu.threads‚17 disk.sda3.in‚0% disk./.usageGB‚1099.6838912963867 disk.nvme0n1.in‚0 disk.nvme0c0n1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.rbd4.out‚0$ memory_percent‚99.95863437652588‰–ÚØž”‚(:“ Ù‹åÔ—˜ª gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.606' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.121199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íe%(•‚(: Ù‹åÔ¿¢ñ disk.dm-1.in‚0 disk.rbd4.out‚0 disk.sda1.out‚0" disk.dm-0.out‚7668.37939453125 network.sent‚ 3.397139e+06 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚66.41552734375 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda.out‚ 60.9140625$ memory_percent‚99.96092319488525' disk.nvme0c0n1.out‚7668.37939453125 disk.sda2.out‚0 disk.dm-1.out‚ 60.9140625 disk.rbd1.in‚0 disk.rbd3.out‚0) proc.memory.percent‚17.47143268585205 cpu‚25.24425625663096 disk.sda2.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 6.40234375 disk.nvme0n1.in‚0% disk.nvme0n1.out‚7668.37939453125 disk.rbd1.out‚ 0.919921875 disk.sda3.out‚ 60.9140625% disk./.usageGB‚1099.7463569641113 disk.sda3.in‚0 disk.dm-0.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 network.recv‚583220) disk./.usagePercent‚73.80490076451632 disk.rbd0.out‚0# proc.memory.rssMB‚ 2862.51953125§‚Ë\2–- _wandbruntime‚1480  _runtime‚1480•®É—à % _timestamp‚1.7884297954346247e+09 _runtime‚1480.139370366 _step‚159 ' lr-AdamW/pg1‚0.00022368802200902954 ' lr-AdamW/pg2‚0.00022368802200902954 trainer/global_step‚1999Å9°§Î˜È % _timestamp‚1.7884297954346247e+09 _runtime‚1480.139370366 _step‚159 ' lr-AdamW/pg1‚0.00022368802200902954 ' lr-AdamW/pg2‚0.00022368802200902954 trainer/global_step‚1999ŸB.Fç2™- _wandbruntime‚1480  _runtime‚1480 MßÖšÐO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125708277910016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116705465284096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1656740422511 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1662201290752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16751452 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1662201290752 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1746621184000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 _step‚160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26249805 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115043281996288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115043281996288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112047451356160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28720376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1662857652224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37518500 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37498464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112047468395520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37518500 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9498353 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28720374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28720374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1656085984763 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_poœ ol.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚116706139648512 trainer/global_step‚1999 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169916416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28720376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127454923034112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127454729177600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1662857652224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9498795 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115043281996288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115043281996288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112047451356160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28720376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1662857652224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37518500 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37498464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112047468395520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37518500 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9498353 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28720374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28720374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1656085984763 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚116706139648512 trainer/global_step‚1999 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169916416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28720376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127454923034112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127454729177600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1662857652224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9498795 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116705465284096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1656740422511 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1662201290752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16751452 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1662201290752 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1746621184000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 _step‚160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26249805 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 Þ·í82œ- _wandbruntime‚1480  _runtime‚1480qh’á&ÛM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112102355486720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1657556857683 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26262549 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9503379 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37516918 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28734392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37536980 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28734390 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113759929383763 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚1480.796975311 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9502929 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66271372 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16759622 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1656902419727 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28734392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1999 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1747479242240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26263001 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125769952587776 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66271372 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28734390 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116763312128512 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115099635035136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1663677093376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127517242181120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125769928647680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112102372526080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1747313533440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37536980 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113759257906447 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116763312128512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115099635035136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115099617032192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884297960915725e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66251308 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66251308 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127517431830016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16759620 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115099617032192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1663677093376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1663020718592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0²Ô]Uæ&žàM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚1480.796975311 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9502929 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66271372 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16759622 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1656902419727 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28734392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1999 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1747479242240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26263001 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125769952587776 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66271372 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28734390 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116763312128512 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115099635035136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1663677093376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127517242181120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125769928647680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112102372526080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1747313533440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37536980 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113759257906447 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116763312128512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115099635035136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115099617032192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884297960915725e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66251308 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66251308 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127517431830016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16759620 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115099617032192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1663677093376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1663020718592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112102355486720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1657556857683 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26262549 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9503379 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37516918 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28734392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37536980 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28734390 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113759929383763 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944¡ªÀtr2Ÿ- _wandbruntime‚1480  _runtime‚1480¡»0—É × % _timestamp‚1.7884297960937366e+09 0loss/FRt1/assignment_loss‚1.9293890124873112 1loss/SRqqt3/assignment_loss‚2.151003505649238 1loss/SRqqt2/detection_loss‚1.7917719155550003 /ûGM”loss/FBt3/detection_loss‚2.8284191712737083 _step‚162 /loss/FRt4/assignment_loss‚4.661051775546784 2loss/SRqqt2/assignment_loss‚1.5538172441224256 /loss/FBt1/detection_loss‚0.9245867790486394 /loss/FBt4/detection_loss‚2.1511164206479276 epoch‚1 /loss/FRt3/assignment_loss‚5.117055797576905 2loss/SRqqt4/assignment_loss‚1.7314589290320874 1loss/SRqqt1/detection_loss‚1.2533803950135525 &loss/total_loss‚52.375937000775295 /loss/FRt1/detection_loss‚1.9025977044774776 2loss/SRqqt1/assignment_loss‚1.1208010549155565 1loss/FBt1/assignment_loss‚0.26779502495666363 0loss/FBt2/assignment_loss‚0.7873871922492981 0loss/FBt4/assignment_loss‚0.7463786133698055 1loss/SRqqt3/detection_loss‚2.5208495187348334 _runtime‚1480.798371973 0loss/FRt2/assignment_loss‚3.0931822035047745 0loss/FBt3/assignment_loss‚0.6387256484478655 /loss/FRt4/detection_loss‚2.2679740139778626 1loss/SRqqt4/detection_loss‚2.1535930544137956 /loss/FRt2/detection_loss‚2.2256721284654404 /loss/FRt3/detection_loss‚3.0701438426971435 .loss/FBt2/detection_loss‚5.487786054611206 trainer/global_step‚1999lªÁCb¡Ü 1loss/FBt1/assignment_loss‚0.26779502495666363 0loss/FBt2/assignment_loss‚0.7873871922492981 0loss/FBt4/assignment_loss‚0.7463786133698055 1loss/SRqqt3/detection_loss‚2.5208495187348334 _runtime‚1480.798371973 0loss/FRt2/assignment_loss‚3.0931822035047745 0loss/FBt3/assignment_loss‚0.6387256484478655 /loss/FRt4/detection_loss‚2.2679740139778626 1loss/SRqqt4/detection_loss‚2.1535930544137956 /loss/FRt2/detection_loss‚2.2256721284654404 /loss/FRt3/detection_loss‚3.0701438426971435 .loss/FBt2/detection_loss‚5.487786054611206 trainer/global_step‚1999 % _timestamp‚1.7884297960937366e+09 0loss/FRt1/assignment_loss‚1.9293890124873112 1loss/SRqqt3/assignment_loss‚2.151003505649238 1loss/SRqqt2/detection_loss‚1.7917719155550003 /loss/FBt3/detection_loss‚2.8284191712737083 _step‚162 /loss/FRt4/assignment_loss‚4.661051775546784 2loss/SRqqt2/assignment_loss‚1.5538172441224256 /loss/FBt1/detection_loss‚0.9245867790486394 /loss/FBt4/detection_loss‚2.1511164206479276 epoch‚1 /loss/FRt3/assignment_loss‚5.117055797576905 2loss/SRqqt4/assignment_loss‚1.7314589290320874 1loss/SRqqt1/detection_loss‚1.2533803950135525 &loss/total_loss‚52.375937000775295 /loss/FRt1/detection_loss‚1.9025977044774776 2loss/SRqqt1/assignment_loss‚1.1208010549155565¢—´Ë-ž¢‚(:“ è‹åÔÛ¥ñ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.606' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.121199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>¥*£‚(:Ÿ è‹åÔåóÌ( disk./.usagePercent‚73.8049880630855 disk.dm-0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 61.45703125 disk.drbd1777.in‚0& proc.memory.availableMB‚ 6.18359375" disk.dm-0.out‚7669.36669921875 disk.sda.out‚ 61.45703125 disk.rbd4.in‚0 disk.nvme0c0n1.in‚0 disk.sda2.in‚0 network.recv‚586573 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚66.49365234375$ memory_percent‚99.96225833892822 network.sent‚ 3.401632e+06 disk.rbd1.out‚ 0.919921875 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0) proc.memory.percent‚17.47143268585205$ disk./.usageGB‚1099.747657775879 disk.sda.in‚0% disk.nvme0n1.out‚7669.36669921875 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2862.51953125 proc.cpu.threads‚17 disk.dm-1.out‚ 61.45703125' disk.nvme0c0n1.out‚7669.36669921875 cpu‚25.2423377651478779€H’¤‚(:‡ ÷‹åÔËÄã gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚91.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—h°Þ¥‚(: ÷‹åÔ–ÉÁ‘ disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 62.2578125 disk.sda.out‚ 62.2578125$ memory_percent‚99.96166229248047 disk.rbd1.out‚ 0.919921875 disk.nvme0c0n1.in‚0% disk./.usageGB‚1099.7507095336914 disk.sda1.in‚0 disk.dm-1.out‚ 62.2578125! disk.nvme0n1.out‚ 7771.8671875 disk.rbd2.in‚0 disk.rbd2.out‚66.49365234375 disk.rbd3.in‚0 disk.sda.in‚0# proc.memory.availableMB‚6.28125 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0# disk.nvme0c0n1.out‚ 7771.8671875 network.recv‚593069 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2862.51953125) proc.memory.percent‚17.47143268585205 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.dm-0.out‚ 7771.8671875 network.sent‚ 3.510318e+06 cpu‚25.240252235771408 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚73.80519286911289 disk.sda2.in‚0©ãœJ“¦‚(:ˆ †ŒåÔœƒØ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.334' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0([)1(§‚(: †ŒåÔlj proc.cpu.threads‚17 network.sent‚ 3.51553e+06' disk.nvme0c0n1.out‚7871.34130859375 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚0) proc.memory.percent‚17.47143268585205 cpu‚25.23844917289302" disk.dm-0.out‚7871.34130859375 disk.dm-1.out‚ 62.82421875 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.sda.out‚ 62.82421875 network.recv‚596899 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0& proc.memory.availableMB‚ 6.37890625% disk.nvme0n1.out‚7871.34130859375 disk.nvme0c0n1.in‚0 disk.sda1.out‚0$ memory_percent‚99.96106624603271) disk./.usagePercent‚73.80519670922591 disk.nvme0n1.in‚0 disk.sda3.out‚ 62.82421875 disk.rbd4.out‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 66.595703125 disk.rbd3.out‚0% disk./.usageGB‚1099.7507667541504 disk.rbd1.out‚ 0.919921875 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2862.51953125!dï 2¨- _wandbruntime‚1516  _runtime‚1516É~”Ç©Á & lr-AdamW/pg1‚0.0002233066539456438 & lr-AdamW/pg2‚0.0002233066539456438 trainer/global_step‚2049 % _timestamp‚1.7884298313114836e+09 _runtime‚1516.016206571 _step‚163Kå#ÌªÆ % _timestamp‚1.7884298313114836e+09 _runtime‚1516.016206571 _step‚163 & lr-AdamW/pg1‚0.0002233066539456438 & lr-AdamW/pg2‚0.0002233066539456438 trainer/global_step‚2049£Uv@2«- _wandbruntime‚1516  _runtime‚1516¶œ¦Þ×'¬ÑO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117860915935744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114792657884160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119564103032320 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117860915935744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1697575761919 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚164 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128790838733312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128790814793216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119564103032320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1703843458048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17159852 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38444552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1789730723328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67865728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚116490250685439 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580392377344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚114792674923520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 trainer/global_step‚2049 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1703187096576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67845690 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29421174 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153139200 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29421176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9728384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38444552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117860933938688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚116489579208331 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119564777396736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0o€^Ü'­ÖO trainer/global_step‚2049 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1703187096576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67845690 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29421174 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153139200 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29421176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9728384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38444552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117860933938688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚116489579208331 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119564777396736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117860915935744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114792657884160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119564103032320 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117860915935744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1697575761919 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚164 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128790838733312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128790814793216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119564103032320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1703843458048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17159852 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38444552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1789730723328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67865728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚116490250685439 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580392377344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚114792674923520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513¤ÉŠÝŸ2®- _wandbruntime‚1516  _runtime‚1516ûÖ£ú¯ÛM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1697737759135 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119621275499008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26900854 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29435192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚2049 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67898224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119621949876736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38442970 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29435190 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1704662899200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38442970 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29435190 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130642888802304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚114847562014720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1698392197091 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119621949876736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67898224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9732832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114847579054080 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67878160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1790595271680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117917286977536 \JDeviceStatsMonitor.on_train_batch_end/alloca_7òRted_bytes.small_pool.allocated‚ 1704662899200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1704006524416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9732382 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26900402 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38463032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116545971251171 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 _runtime‚1516.699596103 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117917268974592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117917268974592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128852489470976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1704006524416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17168020 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1790423271424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚165 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117917286977536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17168022 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128852465530880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119621949876736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38442970 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29435190 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1704662899200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38442970 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29435190 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130642888802304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚114847562014720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1698392197091 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119621949876736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67898224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9732832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114847579054080 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67878160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1790595271680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117917286977536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1704662899200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1704006524416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9732382 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26900402 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38463032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116545971251171 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 _runtime‚1516.699596103 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117917268974592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117917268974592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128852489470976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1704006524416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17168020 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1790423271424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚165 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117917286977536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17168022 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128852465530880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122422740780544 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30121974 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30121974 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39350568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1832864774656 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69472542 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚122423415144960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30121976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30121976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120678567878144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27526397 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17568254 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1551.485410821 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚122423415144960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744172902400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚131873375616512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1738411101327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39350568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27525940 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9957688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131873351676416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117537881451520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚2099 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1744829263872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚122422740780544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117537864412160 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39370604 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133706048631808 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17568254 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1551.485410821 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚122423415144960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.fÕ$Ë›#reed‚ 1744172902400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚131873375616512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1738411101327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39350568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27525940 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9957688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131873351676416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117537881451520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚2099 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1744829263872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚122422740780544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117537864412160 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39370604 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133706048631808 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122422740780544 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30121974 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30121974 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39350568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1832864774656 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69472542 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚122423415144960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30121976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30121976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120678567878144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27526397 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908¨K¸w82¾- _wandbruntime‚1552  _runtime‚1552JÆ®wà&¿ÚM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131935002414080 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9962244 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1833556774912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30135990 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119331341641263 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1739227536499 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9962706 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120734938919936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30135992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚169 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1745648705024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1744992330240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39389084 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1833720386560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1738573098543 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17576422 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122480587624960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119332013118579 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788429867464154e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69525076 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30135990 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27539128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27538664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133768746740736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69505012 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1744992330240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122479913247232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120734920916992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131935026354176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39369022 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚1552.169458875 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39369022 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120734920916992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69505012 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1745648705024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117592768542720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39389084 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27538664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133768746740736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69505012 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1744992330240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122479913247232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120734920916992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131935026354176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39369022 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚1552.169458875 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39369022 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120734920916992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69505012 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1745648705024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117592768542720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39389084 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131935002414080 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9962244 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1833556774912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30135990 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119331341641263 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1739227536499 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9962706 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120734938919936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30135992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚169 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1745648705024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1744992330240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39389084 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1833720386560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1738573098543 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17576422 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122480587624960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119332013118579 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788429867464154e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69525076 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30135990 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414©_ 'þ2Á- _wandbruntime‚1552  _runtime‚1552Ðýž°TÂÎ epoch‚1 _step‚170 2loss/SRqqt1/assignment_loss‚1.2311735513849535 1loss/SRqqt3/assignment_loss‚2.214437929115125 1loss/SRqqt3/detection_loss‚2.6549382912261144 trainer/global_step‚2099 .loss/FRt1/detection_loss‚1.865481511839936 -loss/FRt2/detection_loss‚2.13599779350417 1loss/SRqqt1/detection_loss‚1.2855190230344518 0loss/FRt1/assignment_loss‚2.3097843816203456 .loss/FRt2/assignment_loss‚4.02882719891412 .loss/FRt4/assignment_loss‚3.92555379035861 1loss/SRqqt2/assignment_loss‚1.612205276302263 .loss/FBt4/detection_loss‚2.330677541238921 _runtime‚1552.170926261 1loss/SRqqt4/assignment_loss‚1.655083910403949 /loss/FRt3/detection_loss‚3.7889931574463844 1loss/SRqqt2/detection_loss‚1.8519337416863908 % _timestamp‚1.7884298674662733e+09 2loss/FBt2/assignment_loss‚0.027272882183742315 /loss/FBt2/detection_loss‚6.3984148899714155 /loss/FRt3/assignment_loss‚5.870980083942413 0loss/FBt1/assignment_loss‚0.2962906426895215 0loss/FBt3/assignment_loss‚0.6758949011564255 0loss/FBt4/assignment_loss‚0.7638108283281326 .loss/FRt4/detection_loss‚2.308842829493589 /loss/FBt1/detection_loss‚1.2656340995549993 1loss/SRqqt4/detection_loss‚1.9299671211332645 /loss/FBt3/detection_loss‚2.7134606409817934 %loss/total_loss‚55.14117601751103 £­(YÃÓ 0loss/FRt1/assignment_loss‚2.3097843816203456 .loss/FRt2/assignment_loss‚4.02882719891412 .loss/FRt4/assignment_loss‚3.92555379035861 1loss/SRqqt2/assignment_loss‚1.612205276302263 .loss/FBt4/detection_loss‚2.330677541238921 _runtime‚1552.170926261 1loss/SRqqt4/assignment_loss‚1.655083910403949 /loss/FRt3/detection_loss‚3.7889931574463844 1loss/SRqqt2/detection_loss‚1.8519337416863908 % _timestamp‚1.7884298674662733e+09 2loss/FBt2/assignment_loss‚0.027272882183742315 /loss/FBt2/detection_loss‚6.3984148899714155 /loss/FRt3/assignment_loss‚5.870980083942413 0loss/FBt1/assignment_loss‚0.2962906426895215 0loss/FBt3/assignment_loss‚0.6758949011564255 0loss/FBt4/assignment_loss‚0.7638108283281326 .loss/FRt4/detection_loss‚2.308842829493589 /loss/FBt1/detection_loss‚1.2656340995549993 1loss/SRqqt4/detection_loss‚1.9299671211332645 /loss/FBt3/detection_loss‚2.7134606409817934 %loss/total_loss‚55.14117601751103 epoch‚1 _step‚170 2loss/SRqqt1/assignment_loss‚1.2311735513849535 1loss/SRqqt3/assignment_loss‚2.214437929115125 1loss/SRqqt3/detection_loss‚2.6549382912261144 trainer/global_step‚2099 .loss/FRt1/detection_loss‚1.865481511839936 -loss/FRt2/detection_loss‚2.13599779350417 1loss/SRqqt1/detection_loss‚1.2855190230344518ª›X"’Ä‚(:‡ ³ŒåÔ…ÿú gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚93.901' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øxCn¾Å‚(:— ³ŒåÔÏçî network.recv‚612227# proc.memory.rssMB‚ 2862.52734375 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.drbd1777.in‚0$ memory_percent‚99.96116161346436) disk./.usagePercent‚73.80522384602453 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 cpu‚25.23307580635046 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0& proc.me·ˆÅdmory.availableMB‚ 6.36328125) proc.memory.percent‚17.47148036956787 disk.sda.out‚ 64.71484375% disk./.usageGB‚1099.7511711120605 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 64.71484375 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.dm-0.out‚8070.587890625 disk.rbd2.out‚ 68.248046875 network.sent‚ 3.63593e+06 disk.rbd4.in‚0 disk.sda.in‚0% disk.nvme0c0n1.out‚8070.587890625 disk.sda2.in‚0 disk.sda3.out‚ 64.71484375# disk.nvme0n1.out‚8070.587890625 disk.rbd1.out‚ 0.919921875ú‡êžÆ‚(:“ ÂŒåÔüÇð gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚154.153' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.830599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬%Ç‚(:š ÂŒåÔ€ý«‘$ disk./.usageGB‚1099.751308441162 disk.dm-0.out‚8169.705078125 disk.sda3.out‚ 65.12890625 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875 disk.sda2.out‚0) proc.memory.percent‚17.47148036956787 cpu‚25.231501506420198 disk.rbd2.out‚68.33447265625 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.sda.out‚ 65.12890625 disk.sda1.in‚0 disk.nvme0c0n1.in‚0 disk.sda2.in‚0 network.recv‚617770 disk.sda3.in‚0 disk.rbd4.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚99.96273517608643& proc.memory.availableMB‚ 6.10546875 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 65.12890625 disk.nvme0n1.in‚0% disk.nvme0c0n1.out‚8169.705078125) disk./.usagePercent‚73.80523306229577 disk.sda1.out‚0# disk.nvme0n1.out‚8169.705078125# proc.memory.rssMB‚ 2862.52734375 proc.cpu.threads‚17 network.sent‚ 3.743387e+06 disk.rbd2.in‚0 disk.sda.in‚0Ü›–2È- _wandbruntime‚1587  _runtime‚1587ð¬ÉÉà % _timestamp‚1.7884299023485742e+09 _runtime‚1587.053368608 _step‚171 ' lr-AdamW/pg1‚0.00022247523353641853 ' lr-AdamW/pg2‚0.00022247523353641853 trainer/global_step‚2149ò —ŒÎÊÈ % _timestamp‚1.7884299023485742e+09 _runtime‚1587.053368608 _step‚171 ' lr-AdamW/pg1‚0.00022247523353641853 ' lr-AdamW/pg2‚0.00022247523353641853 trainer/global_step‚2149«¨ìžÜ2Ë- _wandbruntime‚1587  _runtime‚1587¤uÍÙ'ÌÓO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125281378528768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123496237823488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136831949222912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123496219820544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71099394 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30822776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40296656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125281378528768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28164917 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10188263 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122062334420255 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚172 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1785158708224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10187812 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120283070940160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30822776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1785815069696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17976654 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136831757463552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884299023493364e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1779246440735 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40296656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17976652 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123496237823488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134955912499712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30822774 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30822774 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚125282052893184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1785158708224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚1587.054260639 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40276620 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1778592002987 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928\Ñ·Þ'ÍØO % _timestamp‚1.7884299023493364e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1779246440735 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40296656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17976652 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123496237823488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134955912499712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30822774 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30822774 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚125282052893184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1785158708224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚1587.054260639 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40276620 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1778592002987 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125281378528768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123496237823488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136831949222912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123496219820544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71099394 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30822776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40296656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125281378528768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28164917 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10188263 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122062334420255 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚172 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1785158708224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10187812 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120283070940160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30822776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1785815069696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17976654 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136831757463552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944¬4u52Î- _wandbruntime‚1587  _runtime‚1587ãÁb%¨%ÏÝM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71131864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123552572859392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40315136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1876884576256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120337975070720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123552590862336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176194560 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1780062875907 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71131864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40295074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17984822 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123552590862336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71151928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚1587.713431342 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30836790 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28177170 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200134656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136894247678976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123552572859392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40315136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136894447813632 _step‚173 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125338550995456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1786634510848 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1786634510848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10192350 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135017563237376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30836792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17984820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122118054985987 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122117383508671 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135017539297280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1876708381696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1785978136064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2149 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1779408437951 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40295074 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28177634 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10192812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120337992110080 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71151928 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1785978136064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2149 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1779408437951 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40295074 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28177634 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10192812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120337992110080 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71151928 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71131864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123552572859392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40315136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1876884576256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120337975070720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123552590862336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176194560 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1780062875907 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71131864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40295074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17984822 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123552590862336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71151928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚1587.713431342 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30836790 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28177170 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200134656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136894247678976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123552572859392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40315136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136894447813632 _step‚173 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125338550995456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1786634510848 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1786634510848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10192350 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135017563237376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30836792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17984820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122118054985987 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122117383508671 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135017539297280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0­¬ 2Ñ- _wandbruntime‚1587  _runtime‚1587šnïWÒÑ /loss/FRt4/assignment_loss‚4.638034849750753 2loss/SRqqt2/assignment_loss‚1.7861696818360575 1loss/FBt1/assignment_loss‚0.31488527085695395 .loss/FRt1/detection_loss‚2.220938399788879 .loss/FBt2/detection_loss‚4.276652812957764 /loss/FRt1/assignment_loss‚2.223829835653305 0loss/FRt3/assignment_loss‚4.6901490038091485 .loss/FBt1/detection_loss‚1.047333123895737 /loss/FBt3/detection_loss‚2.9305785511221205 _runtime‚1587.714515245 trainer/global_step‚2149 /loss/FRt2/detection_loss‚2.1396505408562145 /loss/FRt4/detection_loss‚2.1725602405411855 1loss/SRqqt2/detection_loss‚1.7159704146561798 1loss/SRqqt4/detection_loss‚1.8105771419953327 2loss/SRqqt3/assignment_loss‚2.1461962930419864 .loss/FRt3/detection_loss‚2.870723160830411 $ _timestamp‚1.788429903009857e+09 0loss/FBt3/assignment_loss‚0.5278314309260038 /loss/SRqqt3/detection_loss‚2.46131313197753 .loss/FBt4/detection_loss‚2.265920005738735 _step‚174 0loss/FRt2/assignment_loss‚3.3467067927122116 2loss/SRqqt1/assignment_loss‚1.3018927179167377 2loss/SRqqt4/assignment_loss‚1.5882877705960858 1loss/FBt2/assignment_loss‚0.04761947877705097 0loss/FBt4/assignment_loss‚0.7272369848215021 0loss/SRqqt1/detection_loss‚1.145522834842696 &loss/total_loss‚50.396580469900584 epoch‚1^!,\ÓÖ /loss/FRt1/assignment_loss‚2.223829835653305 0loss/FRt3/assignment_loss‚4.6901490038091485 .loss/FBt1/detection_loss‚1.047333123895737 /loss/FBt3/detection_loss‚2.9305785511221205 _runtime‚1587.714515245 trainer/global_step‚2149 /loss/FRt2/detection_loss‚2.1396505408562145 /loss/FRt4/detection_loss‚2.1725602405411855 1loss/SRqqt2/detection_loss‚1.7159704146561798 1loss/SRqqt4/detection_loss‚1.8105771419953327 2loss/SRqqt3/assignment_loss‚2.1461962930419864 .loss/FRt3/detection_loss‚2.870723160830411 $ _timestamp‚1.788429903009857e+09 0loss/FBt3/assignment_loss‚0.5278314309260038 /loss/SRqqt3/detection_loss‚2.46131313197753 .loss/FBt4/detection_loss‚2.265920005738735 _step‚174 0loss/FRt2/assignment_loss‚3.3467067927122116 2loss/SRqqt1/assignment_loss‚1.3018927179167377 2loss/SRqqt4/assignment_loss‚1.5882877705960858 1loss/FBt2/assignment_loss‚0.04761947877705097 0loss/FBt4/assignment_loss‚0.7272369848215021 0loss/SRqqt1/detection_loss‚1.145522834842696 &loss/total_loss‚50.396580469900584 epoch‚1 /loss/FRt4/assignment_loss‚4.638034849750753 2loss/SRqqt2/assignment_loss‚1.7861696818360575 1loss/FBt1/assignment_loss‚0.31488527085695395 .loss/FRt1/detection_loss‚2.220938399788879 .loss/FBt2/detection_loss‚4.276652812957764®ÞèúÒ“Ô‚(:ˆ ÑŒåÔ³ÿ€Ž gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.629' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9258 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÈ&Ë#Õ‚(:˜ ÑŒåÔ›æÏ disk.sda3.in‚0 disk.sda3.out‚ 65.62890625 disk.rbd3.in‚0 disk.dm-0.in‚0 disk.sda2.out‚0) proc.memory.percent‚17.47148036956787 disk.dm-0.out‚8268.794921875 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.rbd4.out‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0 network.sent‚ 3.7496e+06 disk.rbd4.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0# disk.nvme0n1.out‚8268.794921875 disk.rbd1.in‚0 cpu‚25.22968513931268 disk.rbd1.out‚ 0.919921875 disk.rbd0.in‚0) disk./.usagePercent‚73.80522794214508% disk./.usageGB‚1099.7512321472168 disk.drbd1777.out‚0 disk.sda.in‚0 disk.sda.out‚ 65.62890625 disk.dm-1.out‚ 65.62890625% disk.nvme0c0n1.out‚8268.794921875 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.rbd2.out‚68.35009765625$ memory_percent‚99.96459484100342& proc.memory.availableMB‚ 5.80078125# proc.memory.rssMB‚ 2862.52734375 network.recv‚621865 proc.cpu.threads‚17’éTÌžÖ‚(:“ àŒåÔ·Œ—Ž gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.801' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.760199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“~ð,ׂ(:¡ àŒåÔÒ®°) disk./.usagePercent‚73.80523997449919 disk.dm-0.in‚0" disk.dm-0.out‚8368.71044921875 disk.sda.out‚ 66.51953125% disk.nvme0n1.out‚8368.71044921875 disk.drbd1777.out‚0 disk.rbd4.out‚0# proc.memory.rssMB‚ 2862.52734375 network.sent‚ 3.857833e+06 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 66.51953125' disk.nvme0c0n1.out‚8368.71044921875) proc.memory.percent‚17.47148036956787 disk.rbd0.in‚0 disk.rbd2.out‚68.41259765625 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0 cpu‚25.228040406625585 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 66.51953125 network.recv‚628733% disk./.usageGB‚1099.7514114379883 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd3.in‚0$ memory_percent‚99.96030330657959 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875 disk.rbd0.out‚0 disk.drbd1777.in‚0& proc.memory.availableMB‚ 6.50390625z3°,žØ‚(:“ ïŒåÔ¢Ý gpu.0.gpu‚42.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.661' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.132199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Yí_š*Ù‚(:Ÿ ïŒåÔÚéŒ disk.rbd4.in‚0 disk.rbd4.out‚0" disk.dm-0.out‚8467.73388671875 disk.nvme0c0n1.in‚0 network.sent‚ 3.864214e+06 disk.sda3.out‚ 67.14453125' disk.nvme0c0n1.out‚8467.73388671875$ memory_percent‚99.96180534362793# proc.memory.rssMB‚ 2862.52734375% disk./.usageGB‚1099.7514114379883) disk./.usagePercent‚73.80523997449919 disk.nvme0n1.in‚0 disk.rbd1.out‚ 0.919921875) proc.memory.percent‚17.47148036956787 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚0 cpu‚25.22642687730835 disk.sda3.in‚0 disk.sda.out‚ 67.14453125 disk.sda1.out‚0 disk.dm-1.out‚ 67.14453125 disk.rbd1.in‚0% proc.memory.availableMB‚ 6.2578125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚8467.73388671875 network.recv‚633557 disk.rbd2.out‚69.93994140625 disk.drbd1777.in‚0ÎÈ~2Ú- _wandbruntime‚1622  _runtime‚1622‘dNÈÛ trainer/global_step‚2199 $ _timestamp‚1.788429937785995e+09 _runtime‚1622.490788622 _step‚175 ' lr-AdamW/pg1‚0.00022202535445734609 ' lr-AdamW/pg2‚0.00022202535445734609v)·<ÍÜÇ ' lr-AdamW/pg2‚0.00022202535445734609 trainer/global_step‚2199 $ _timestamp‚1.788429937785995e+09 _runtime‚1622.490788622 _step‚175 ' lr-AdamW/pg1‚0.00022202535445734609¯5 Š2Ý- _wandbruntime‚1622  _runtime‚1622 ‚ˆÙ'ÞÓO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10415833 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1819427342395 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1826144514048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72746284 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1918870112768 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826800875520 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41222708 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28800887 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28800443 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123028294507520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18385052 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128140690641408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31523576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1919046320640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41202672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128140016276992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2199 _runtime‚1622.491696552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41222708 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128140690641408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139957295555584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚176 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126313889765888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1826144514048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10415391 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123028277468160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126313871762944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31523574 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139957495703552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72726246 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128140016276992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138038425442816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126313889765888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚124848376287663 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41202672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430GOyógßØO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41202672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128140016276992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2199 _runtime‚1622.491696552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41222708 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128140690641408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139957295555584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚176 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126313889765888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1826144514048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10415391 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123028277468160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126313871762944 PCDeviceSt±å[0watsMonitor.on_train_batch_start/allocation.large_pool.freed‚31523574 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139957495703552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72726246 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128140016276992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138038425442816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126313889765888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚124848376287663 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41202672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10415833 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1819427342395 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1826144514048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72746284 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1918870112768 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826800875520 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41222708 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28800887 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28800443 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123028294507520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18385052 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128140690641408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31523576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1919046320640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0°¡”‰”2à- _wandbruntime‚1623  _runtime‚1623ýûã&áÝM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827620316672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28813627 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41221126 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128197863121408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128197863121408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1826963941888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1919732996096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1820898215315 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41241188 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31537592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41221126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123083198638080 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72778780 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138100076180480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31537590 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138100100120576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28813188 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123083181598720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72758716 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18393220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31537592 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884299384578757e+09 _runtime‚1623.163112124 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126370242804736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126370224801792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72758716 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919896607744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚2199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10419968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚177 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31537590 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41241188 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚124904096853395 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126370242804736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124903425376079 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1827620316672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1820243777359 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827620316672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28813627 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41221126 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128197863121408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128197863121408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1826963941888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1919732996096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1820898215315 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41241188 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31537592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41221126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123083198638080 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72778780 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138100076180480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31537590 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138100100120576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28813188 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123083181598720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72758716 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18393220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31537592 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884299384578757e+09 _runtime‚1623.163112124 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126370242804736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126370224801792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72758716 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919896607744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚2199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10419968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚177 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31537590 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41241188 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚124904096853395 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126370242804736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124903425376079 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1827620316672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1820243777359 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ôNÏü$ 1867786681344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129131523705344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884299733769379e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1860917119551 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74353098 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1962038626304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127633746677963 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74373136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42128724 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143083024892416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141120962326016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129131541708288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚2249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32224374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1867130319872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚130999328389632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143082843618816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125773501035520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29439617 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29440056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18793452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10646602 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32224376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127634418155071 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42128724 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1961881292800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10646165 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129131523705344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129131541708288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18793454 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125773483996160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130998654025216 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1867786681344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157333504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚130999328389632 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130998654025216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42148760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 _step‚180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32224374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1860262681803 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 1658.08187259 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130998654025216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42148760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 _step‚180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32224374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1860262681803 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 1658.08187259 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1867786681344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129131523705344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884299733769379e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1860917119551 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74353098 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1962038626304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127633746677963 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74373136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42128724 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143083024892416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141120962326016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129131541708288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚2249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32224374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1867130319872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚130999328389632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143082843618816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125773501035520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29439617 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29440056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18793452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10646602 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32224376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127634418155071 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42128724 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1961881292800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10646165 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129131523705344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129131541708288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18793454 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125773483996160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130998654025216´³02ð- _wandbruntime‚1658  _runtime‚1658æ­ã&ñÝM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129187876744192 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129187876744192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18801620 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10650700 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125828388126720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32238392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131055826491904 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74385568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18801622 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚1658.746785067 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1962752381440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125828405166080 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29452320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42167240 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42147178 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1867949747712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32238390 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141182613063680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129187894747136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29452775 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131056500869632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127690138720803 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143145546705408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131056500869632 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129187894747136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74405632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74385568 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32238390 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1962909701632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884299740414166e+09Šâï òâM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29452775 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131056500869632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127690138720803 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143145546705408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131056500869632 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129187894747136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74405632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74385568 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32238390 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1962909701632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884299740414166e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129187876744192 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129187876744192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18801620 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10650700 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125828388126720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32238392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131055826491904 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74385568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18801622 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚181 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚1658.746785067 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1962752381440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125828405166080 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29452320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42167240 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42147178 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1867949747712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32238390 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141182613063680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129187894747136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2µÇm©Å2ó- _wandbruntime‚1658  _runtime‚1658)v)ÕWôÑ 2loss/SRqqt1/assignment_loss‚1.2107415680578093 /loss/FRt3/detection_loss‚2.9907853641817645 epoch‚1 $ _timestamp‚1.788429974043286e+09 trainer/global_step‚2249 0loss/FRt1/assignment_loss‚2.4101480270425477 2loss/SRqqt3/assignment_loss‚2.2502826578689343 2loss/SRqqt4/assignment_loss‚2.0111601933470515 0loss/FBt3/assignment_loss‚0.6349786961320205 /loss/FRt1/detection_loss‚2.0333886674294868 1loss/SRqqt1/detection_loss‚1.1518639125573782 .loss/FBt3/detection_loss‚3.839781868457794 /loss/FBt4/detection_loss‚1.7676262239615121 %loss/total_loss‚53.73060898270555 0loss/FRt2/assignment_loss‚3.8933674389002273 /loss/FRt4/assignment_loss‚4.451725556736901 /loss/FBt4/assignment_loss‚0.851172094543775 .loss/FRt4/detection_loss‚2.154640887464796 1loss/SRqqt2/detection_loss‚2.0554424622591507 1loss/FBt1/assignment_loss‚0.31186289956391766 .loss/FRt2/detection_loss‚2.571834389044314 1loss/SRqqt4/detection_loss‚1.9039914160966873 .loss/FBt2/detection_loss‚4.475567837556203 _step‚182 /loss/FRt3/assignment_loss‚5.033364980451522 0loss/FBt2/assignment_loss‚0.4382791516885239 2loss/SRqqt2/assignment_loss‚1.8972480369216584 0loss/SRqqt3/detection_loss‚2.537633298924475 /loss/FBt1/detection_loss‚0.8537213535171014 _runtime‚1658.747924614ù·bñ\õÖ 2loss/SRqqt3/assignment_loss‚2.2502826578689343 2loss/SRqqt4/assignment_loss‚2.0111601933470515 0loss/FBt3/assignment_loss‚0.6349786961320205 /loss/FRt1/detection_loss‚2.0333886674294868 1loss/SRqqt1/detection_loss‚1.1518639125573782 .loss/FBt3/detection_loss‚3.839781868457794 /loss/FBt4/detection_loss‚1.7676262239615121 %loss/total_loss‚53.73060898270555 0loss/FRt2/assignment_loss‚3.8933674389002273 /loss/FRt4/assignment_loss‚4.451725556736901 /loss/FBt4/assignment_loss‚0.851172094543775 .loss/FRt4/detection_loss‚2.154640887464796 1loss/SRqqt2/detection_loss‚2.0554424622591507 1loss/FBt1/assignment_loss‚0.31186289956391766 .loss/FRt2/detection_loss‚2.571834389044314 1loss/SRqqt4/detection_loss‚1.9039914160966873 .loss/FBt2/detection_loss‚4.475567837556203 _step‚182 /loss/FRt3/assignment_loss‚5.033364980451522 0loss/FBt2/assignment_loss‚0.4382791516885239 2loss/SRqqt2/assignment_loss‚1.8972480369216584 0loss/SRqqt3/detection_loss‚2.537633298924475 /loss/FBt1/detection_loss‚0.8537213535171014 _runtime‚1658.747924614 2loss/SRqqt1/assignment_loss‚1.2107415680578093 /loss/FRt3/detection_loss‚2.9907853641817645 epoch‚1 $ _timestamp‚1.788429974043286e+09 trainer/global_step‚2249 0loss/FRt1/assignment_loss‚2.4101480270425477¶c©É$“ö‚(:ˆ œåÔ“³ü gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚136.314' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.2628 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}%žë÷‚(: œåÔ¯ disk.sda.in‚0 disk.nvme0n1.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚ 8666.703125 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda.out‚ 68.74609375 proc.cpu.threads‚17) disk./.usagePercent‚73.80528810391563 disk.rbd1.out‚ 0.919921875 disk.rbd3.in‚0 disk.dm-1.in‚0 network.recv‚647301" disk.nvme0c0n1.out‚ 8666.703125 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 68.74609375$ proc.memory.availableMB‚6.140625 disk.nvme0n1.out‚ 8666.703125 disk.rbd2.out‚ 70.2294921875 disk.drbd1777.in‚0$ memory_percent‚99.96252059936523# proc.memory.rssMB‚ 2862.52734375 cpu‚25.221723344310117 disk.nvme0c0n1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 68.74609375 disk.rbd4.out‚0 disk.sda1.out‚0) proc.memory.percent‚17.47148036956787 network.sent‚ 3.983401e+06 disk.sda2.out‚0 disk.dm-0.in‚0 disk.sda1.in‚0% disk./.usageGB‚1099.7521286010742 disk.sda2.in‚0 disk.drbd1777.out‚0 ËÅâžø‚(:“ «åÔ‰œù gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.080399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àq·“ù‚(: «åÔꑘ% disk./.usageGB‚1099.7521476745605 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875# disk.nvme0c0n1.out‚ 8765.7734375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚0 disk.sda.out‚ 69.70703125 disk.rbd2.in‚0! disk.nvme0n1.out‚ 8765.7734375 disk.sda3.out‚ 69.70703125 disk.drbd1777.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd2.out‚ 70.2412109375( disk./.usagePercent‚73.8052893839533$ memory_percent‚99.96099472045898 proc.cpu.threads‚17 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.rbd0.out‚0$ proc.memory.availableMB‚6.390625# proc.memory.rssMB‚ 2862.52734375 disk.dm-0.out‚ 8765.7734375 disk.nvme0c0n1.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.47148036956787 network.sent‚ 4.092023e+06 disk.sda1.in‚0 disk.dm-1.out‚ 69.70703125 disk.nvme0n1.in‚0 cpu‚25.22021915262713 disk.sda1.out‚0 network.recv‚653971û/Õ2ú- _wandbruntime‚1693  _runtime‚1693Ûš)–Éûà % _timestamp‚1.7884300087262974e+09 _runtime‚1693.431045944 _step‚183 ' lr-AdamW/pg1‚0.00022105772730137232 ' lr-AdamW/pg2‚0.00022105772730137232 trainer/global_step‚2299V]lÎüÈ trainer/global_step‚2299 % _timestamp‚1.7884300087262974e+09 _runtime‚1693.431045944 _step‚183 ' lr-AdamW/pg1‚0.00022105772730137232 ' lr-AdamW/pg2‚0.00022105772730137232·îbÏ2ý- _wandbruntime‚1693  _runtime‚1693ãËòÙ'þÓO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10874509 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43074812 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30076361 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131949175647744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133857291773440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 _runtime‚1693.432338925 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10874947 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146208533715968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131949193650688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133857291773440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1908772487168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1908116125696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32925176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75979950 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43074812 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133857966137856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚2299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133857966137856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32925176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43054776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1908772487168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130420460022479 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32925174 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128518690524160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32925174 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128518707563520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1901752458959 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131949193650688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75999988 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144203523149312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43054776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19201852 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133857291773440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 _runtime‚1693.432338925 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10874947 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146208533715968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131949193650688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133857291773440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1908772487168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1908116125696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32925176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75979950 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43074812 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133857966137856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚2299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133857966137856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32925176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43054776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1908772487168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130420460022479 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32925174 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128518690524160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32925174 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128518707563520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1901752458959 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131949193650688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75999988 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144203523149312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43054776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.pâ`Xeak‚24399 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19201852 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32939190 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76012420 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 _runtime‚1694.106466838 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43093292 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚132005546689536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76032484 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130475509110895 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2005700208640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130476180588211 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128573594654720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30089045 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30089504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1909591928320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76012420 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ãh¬Êè&‚âM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130475509110895 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2005700208640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130476180588211 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128573594654720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30089045 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30089504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1909591928320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76012420 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10879025 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2005865917440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43073230 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133915138617856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19210020 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32939192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133914464240128 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1909591928320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚132005546689536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10879482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144265149946880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128573611694080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884300094013684e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144265173886976 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚132005528686592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚2299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32939190 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76012420 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 _runtime‚1694.106466838 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43093292 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚132005546689536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76032484 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¹À`]2ƒ- _wandbruntime‚1694  _runtime‚1694Ô4ƒÉV„Ð .loss/FBt1/detection_loss‚0.709575823017442 /loss/FBt3/detection_loss‚2.8794572005669274 2loss/SRqqt2/assignment_loss‚1.8384841147065163 0loss/FBt3/assignment_loss‚0.6948904580203816 1loss/SRqqt2/detection_loss‚2.0412006986141207 %loss/total_loss‚51.80452941765064 0loss/FRt1/assignment_loss‚1.8741584957814683 /loss/FRt3/assignment_loss‚4.779821115991344 0loss/FBt2/assignment_loss‚1.0989653885364532 /loss/FRt2/detection_loss‚2.2101095200971117 2loss/SRqqt4/assignment_loss‚1.6754532290250064 /loss/FBt2/detection_loss‚3.9173731803894043 1loss/SRqqt1/detection_loss‚1.1961274849018082 trainer/global_step‚2299 /loss/FRt2/assignment_loss‚4.314248470372932 0loss/FRt4/assignment_loss‚3.4630582976341246 /loss/FRt3/detection_loss‚3.0948819492174233 /loss/FRt4/detection_loss‚2.1952645707130434 _runtime‚1694.107462971 2loss/SRqqt1/assignment_loss‚0.9980045585834887 1loss/SRqqt3/assignment_loss‚2.250993718703588 1loss/SRqqt3/detection_loss‚2.5069714029630026 -loss/FBt4/detection_loss‚2.72543987902728 epoch‚1 _step‚186 /loss/FBt4/assignment_loss‚0.925125162709843 /loss/FRt1/detection_loss‚2.0906850609417056 1loss/SRqqt4/detection_loss‚2.0382244288921356 # _timestamp‚1.78843000940286e+09 0loss/FBt1/assignment_loss‚0.2860152082440878,˜oÌ[…Õ 0loss/FBt2/assignment_loss‚1.0989653885364532 /loss/FRt2/detection_loss‚2.2101095200971117 2loss/SRqqt4/assignment_loss‚1.6754532290250064 /loss/FBt2/detection_loss‚3.9173731803894043 1loss/SRqqt1/detection_loss‚1.1961274849018082 trainer/global_step‚2299 /loss/FRt2/assignment_loss‚4.314248470372932 0loss/FRt4/assignment_loss‚3.4630582976341246 /loss/FRt3/detection_loss‚3.0948819492174233 /loss/FRt4/detection_loss‚2.1952645707130434 _runtime‚1694.107462971 2loss/SRqqt1/assignment_loss‚0.9980045585834887 1loss/SRqqt3/assignment_loss‚2.250993718703588 1loss/SRqqt3/detection_loss‚2.5069714029630026 -loss/FBt4/detection_loss‚2.72543987902728 epoch‚1 _step‚186 /loss/FBt4/assignment_loss‚0.925125162709843 /loss/FRt1/detection_loss‚2.0906850609417056 1loss/SRqqt4/detection_loss‚2.0382244288921356 # _timestamp‚1.78843000940286e+09 0loss/FBt1/assignment_loss‚0.2860152082440878 .loss/FBt1/detection_loss‚0.709575823017442 /loss/FBt3/detection_loss‚2.8794572005669274 2loss/SRqqt2/assignment_loss‚1.8384841147065163 0loss/FBt3/assignment_loss‚0.6948904580203816 1loss/SRqqt2/detection_loss‚2.0412006986141207 %loss/total_loss‚51.80452941765064 0loss/FRt1/assignment_loss‚1.8741584957814683 /loss/FRt3/assignment_loss‚4.779821115991344ºA÷·’†‚(:‡ ºåÔá“Ž gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.579' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.5158 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñØ‘)'‡‚(:œ ºåÔÍéñ network.sent‚ 4.097146e+06) disk./.usagePercent‚73.80529066399097 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 5.60546875) proc.memory.percent‚17.47148036956787 disk.sda3.in‚0 disk.sda3.out‚ 70.16015625 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 71.8779296875 disk.dm-1.out‚ 70.16015625 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda.in‚0$ disk.nvme0n1.out‚8766.2490234375 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚8766.2490234375# proc.memory.rssMB‚ 2862.52734375 proc.cpu.threads‚17 disk.dm-0.in‚0 cpu‚25.21871408693979! disk.dm-0.out‚8766.2490234375 disk.sda1.in‚0 disk.rbd3.out‚0 network.recv‚657413 disk.drbd1777.in‚0 disk.sda.out‚ 70.16015625% disk./.usageGB‚1099.7521667480469 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd1.out‚ 0.919921875$ memory_percent‚99.96578693389893eVŠ«“ˆ‚(:ˆ ÉåÔÇðÇ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.039' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓGôÄ#‰‚(:˜ ÉåÔæÇû disk.sda2.in‚0 disk.rbd4.in‚0$ memory_percent‚99.96581077575684& disk.nvme0c0n1.out‚8865.3740234375 disk.rbd4.out‚0 cpu‚25.217253910436128( disk./.usagePercent‚73.8052893839533 disk.dm-0.in‚0! disk.dm-0.out‚8865.3740234375 disk.sda.in‚0$ disk.nvme0n1.out‚8865.3740234375 disk.sda3.out‚ 70.7265625 network.recv‚664220 disk.rbd1.out‚ 0.919921875 disk.rbd0.out‚0 disk.rbd2.out‚ 71.8935546875 disk.drbd1777.out‚0 network.sent‚ 4.206079e+06 disk.dm-1.out‚ 70.7265625 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 70.7265625 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0% proc.memory.availableMB‚ 5.6015625) proc.memory.percent‚17.47148036956787% disk./.usageGB‚1099.7521476745605 disk.sda1.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2862.52734375 proc.cpu.threads‚17 disk.dm-1.in‚0ç-ŸÂŠ‚(:’ ØåÔóð gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.19' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.037999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®D¾?#‹‚(:˜ ØåÔý‘ disk.sda1.in‚0( disk./.usagePercent‚73.8052893839533 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 network.sent‚ 4.211583e+06% disk./.usageGB‚1099.7521476745605 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.out‚ 71.4296875 disk.drbd1777.out‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚8964.69970703125 disk.sda2.out‚0 disk.drbd1777.in‚0$ memory_percent‚99.96681213378906# proc.memory.rssMB‚ 2862.52734375) proc.memory.percent‚17.47148036956787 proc.cpu.threads‚17 disk.sda.out‚ 71.4296875 disk.nvme0c0n1.in‚0 disk.rbd2.out‚ 72.0185546875% disk.nvme0n1.out‚8964.69970703125 disk.rbd1.out‚ 0.919921875 disk.rbd0.out‚0 network.recv‚668315 disk.dm-0.in‚0" disk.dm-0.out‚8964.69970703125 disk.nvme0n1.in‚0 disk.rbd2.in‚0 cpu‚25.215820365698445 disk.dm-1.out‚ 71.4296875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0" proc.memory.availableMB‚5.4375Lá¢Ø2Œ- _wandbruntime‚1728  _runtime‚1728_)@’ÇÁ ' lr-AdamW/pg1‚0.00022054018087651995 ' lr-AdamW/pg2‚0.00022054018087651995 trainer/global_step‚2349 $ _timestamp‚1.788430044143857e+09 _runtime‚ 1728.84858711 _step‚187àcÂÜÌŽÆ _runtime‚ 1728.84858711 _step‚187 ' lr-AdamW/pg1‚0.00022054018087651995 ' lr-AdamW/pg2‚0.00022054018087651995 trainer/global_step‚2349 $ _timestamp‚1.788430044143857e+09»8–;’2- _wandbruntime‚1728  _runtime‚1728q“:a$ÓO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77626840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198050816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚2349 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44000864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33625974 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134766827590144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147286036092416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1949758292992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚188 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44000864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚136715929521664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134766845593088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11104175 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2047971559424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2047797448704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1942587798367 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚1728.849597434 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33625976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19610254 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149334031591936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174110720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚136716603886080 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33625976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134766845593088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1949101931520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147286060032512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33625974 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136715929521664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024çrEÞ'‘ØO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33625976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134766845593088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1949101931520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147286060032512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33625974 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136715929521664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77626840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198050816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚2349 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44000864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33625974 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134766827590144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147286036092416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1949758292992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚188 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44000864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚136715929521664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134766845593088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11104175 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2047971559424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2047797448704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1942587798367 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚1728.849597434 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33625976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19610254 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149334031591936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174110720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚136716603886080¼µ¨f2’- _wandbruntime‚1729  _runtime‚1729×½­Wã&“ÝM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33639992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30727125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147347686830080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77639272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19618420 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1949921359360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33639990 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44019344 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134823180628992 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134823198631936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44019344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149396532016640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33639990 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131318818222080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33639992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1943404233539 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149396344464896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19618422 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134823198631936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77659336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131318801182720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136773776366080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30727574 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43999282 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77659336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1942749795583 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚8447>Õ’5”âM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134823198631936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44019344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149396532016640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33639990 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131318818222080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33639992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1943404233539 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149396344464896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19618422 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134823198631936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77659336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131318801182720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136773776366080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30727574 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43999282 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77659336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1942749795583 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33639992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30727125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147347686830080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77639272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19618420 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1949921359360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33639990 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44019344 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134823180628992½ìtÂ|2•- _wandbruntime‚1729  _runtime‚1729>{sáU–Ï 2loss/SRqqt4/assignment_loss‚1.5640987263371546 /loss/FBt1/detection_loss‚0.9016664224992098 %loss/total_loss‚50.74036056637542 % _timestamp‚1.7884300448042276e+09 epoch‚1 _step‚190 1loss/SRqqt1/detection_loss‚1.0900484500170657 1loss/SRqqt4/detection_loss‚1.9598707989567803 /loss/FRt3/assignment_loss‚4.797789844599637 2loss/SRqqt1/assignment_loss‚1.6188491644394216 .loss/FRt4/detection_loss‚2.125995915110518 2loss/SRqqt2/assignment_loss‚1.7876760049358658 0loss/FBt1/assignment_loss‚0.2223945614738293 .loss/FBt2/detection_loss‚4.078882013048444 /loss/FBt3/detection_loss‚3.3308749318122866 -loss/FRt2/detection_loss‚2.33860292478844 .loss/FRt1/assignment_loss‚2.41625476005126 /loss/FRt4/assignment_loss‚3.163897133455044 1loss/FBt2/assignment_loss‚0.03739293478422821 0loss/FBt3/assignment_loss‚0.4936458384611342 1loss/SRqqt2/detection_loss‚1.9326541715342065 0loss/SRqqt3/detection_loss‚2.491723951366213 0loss/FRt2/assignment_loss‚4.0492825552269265 2loss/SRqqt3/assignment_loss‚2.3550877740813627 0loss/FBt4/assignment_loss‚0.6999989889169994 .loss/FRt1/detection_loss‚1.938468063668329 .loss/FRt3/detection_loss‚3.356763395396146 /loss/FBt4/detection_loss‚1.9884412414149235 trainer/global_step‚2349 _runtime‚1729.508907353)¢‘ýZ—Ô 1loss/SRqqt1/detection_loss‚1.0900484500170657 1loss/SRqqt4/detection_loss‚1.9598707989567803 /loss/FRt3/assignment_loss‚4.797789844599637 2loss/SRqqt1/assignment_loss‚1.6188491644394216 .loss/FRt4/detection_loss‚2.125995915110518 2loss/SRqqt2/assignment_loss‚1.7876760049358658 0loss/FBt1/assignment_loss‚0.2223945614738293 .loss/FBt2/detection_loss‚4.078882013048444 /loss/FBt3/detection_loss‚3.3308749318122866 -loss/FRt2/detection_loss‚2.33860292478844 .loss/FRt1/assignment_loss‚2.41625476005126 /loss/FRt4/assignment_loss‚3.163897133455044 1loss/FBt2/assignment_loss‚0.03739293478422821 0loss/FBt3/assignment_loss‚0.4936458384611342 1loss/SRqqt2/detection_loss‚1.9326541715342065 0loss/SRqqt3/detection_loss‚2.491723951366213 0loss/FRt2/assignment_loss‚4.0492825552269265 2loss/SRqqt3/assignment_loss‚2.3550877740813627 0loss/FBt4/assignment_loss‚0.6999989889169994 .loss/FRt1/detection_loss‚1.938468063668329 .loss/FRt3/detection_loss‚3.356763395396146 /loss/FBt4/detection_loss‚1.9884412414149235 trainer/global_step‚2349 _runtime‚1729.508907353 2loss/SRqqt4/assignment_loss‚1.5640987263371546 /loss/FBt1/detection_loss‚0.9016664224992098 %loss/total_loss‚50.74036056637542 % _timestamp‚1.7884300448042276e+09 epoch‚1 _step‚190¾k–A‘˜‚(:† çåÔ‡¢è gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.97' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˸™$™‚(:™ çåÔ¹¥É‘ disk.drbd1777.out‚0 disk.dm-1.out‚ 71.98828125$ disk./.usageGB‚1099.752212524414& disk.nvme0c0n1.out‚9063.8505859375 disk.sda3.out‚ 71.98828125 disk.dm-1.in‚0 disk.rbd1.out‚ 0.919921875) proc.memory.percent‚17.47152805328369 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.rbd1.in‚0$ proc.memory.availableMB‚5.421875 disk.sda3.in‚0 disk.rbd4.in‚0 disk.dm-0.in‚0! disk.dm-0.out‚9063.8505859375 disk.nvme0n1.in‚0 disk.rbd0.out‚0# memory_percent‚99.9669075012207 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda.out‚ 71.98828125 disk.sda1.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 72.0341796875 disk.rbd3.in‚0# proc.memory.rssMB‚ 2862.53515625 cpu‚25.214395285223414 network.recv‚673079) disk./.usagePercent‚73.80529373608138 disk.rbd4.out‚0$ disk.nvme0n1.out‚9063.8505859375 network.sent‚ 4.217905e+06 disk.nvme0c0n1.in‚0eDu–š‚(:’ öåÔÈýü gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.98' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.995999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«|ü›‚(:’ öåÔ¯àÈ disk.dm-0.in‚0 disk.sda.out‚72.65625 disk.sda1.in‚0$ memory_percent‚99.96740818023682 network.recv‚678720 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0% disk.nvme0c0n1.out‚9163.541015625 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 cpu‚25.213039751978375 network.sent‚ 4.325281e+06 disk.rbd0.in‚0 disk.rbd4.out‚0& proc.memory.availableMB‚ 5.33984375 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-0.out‚9163.541015625# disk.nvme0n1.out‚9163.541015625 disk.rbd1.out‚ 0.919921875) proc.memory.percent‚17.47152805328369 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.out‚72.65625# proc.memory.rssMB‚ 2862.53515625) disk./.usagePercent‚73.80530218433002% disk./.usageGB‚1099.7523384094238 disk.rbd2.out‚72.13232421875 disk.sda3.out‚72.65625Nì2œ- _wandbruntime‚1764  _runtime‚1764Ø=Jľ _step‚191 % lr-AdamW/pg1‚0.000220000288135151 % lr-AdamW/pg2‚0.000220000288135151 trainer/global_step‚2399 $ _timestamp‚1.788430079611304e+09 _runtime‚1764.31606544198~ ɞà % lr-AdamW/pg1‚0.000220000288135151 % lr-AdamW/pg2‚0.000220000288135151 trainer/global_step‚2399 $ _timestamp‚1.788430079611304e+09 _runtime‚1764.316065441 _step‚191¿:ú‘¦2Ÿ- _wandbruntime‚1764  _runtime‚1764IÛhÀÙ' ÓO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137584497535488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137584479532544 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2090956748800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139575241634304 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152459719386624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134009103580160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34326774 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137584497535488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79233654 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135991872280187 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚1764.317007666 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20018652 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31353286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139574567269888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1990744098816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1990087737344 trainer/global_step‚2399 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1990087737344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34326776 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44926916 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1990744098816 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137584479532544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2091122470912 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44906880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139575241634304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79253692 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31352831 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11334179 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44926916 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44906880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150368572975616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1983423137775 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1982768700027 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 % _timestamp‚1.7884300796120327e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34326774 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0œ]áÞ'¡ØO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137584479532544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2091122470912 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44906880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139575241634304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79253692 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31352831 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11334179 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44926916 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44906880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150368572975616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1983423137775 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1982768700027 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 % _timestamp‚1.7884300796120327e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34326774 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137584497535488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137584479532544 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2090956748800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139575241634304 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152459719386624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134009103580160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34326774 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137584497535488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79233654 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135991872280187 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚1764.317007666 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20018652 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31353286 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139574567269888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1990744098816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1990087737344 trainer/global_step‚2399 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1990087737344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34326776 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44926916 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1990744098816 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605À~llª2¢- _wandbruntime‚1764  _runtime‚1764{j%ú³ £ÝM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11339255 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136048264323027 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20026820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137640850574336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2091818020352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137640850574336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79286188 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34340792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134064024750080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚193 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139632414114304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31366077 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _KDevie˜èm0ceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150430223713280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134064007710720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79266124 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136047592845711 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1990907165184 _runtime‚1764.994854821 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884300802895396e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2091981632000 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1991563539968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150430247653376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79286188 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34340790 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31365623 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139632414114304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1984239572947 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44945396ÛŽC$è&¤âM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1991563539968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150430247653376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79286188 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34340790 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31365623 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139632414114304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1984239572947 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44945396 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11339255 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136048264323027 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20026820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137640850574336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2091818020352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137640850574336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79286188 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34340792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134064024750080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚193 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139632414114304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31366077 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150430223713280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134064007710720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79266124 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136047592845711 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1990907165184 _runtime‚1764.994854821 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884300802895396e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2091981632000Á…­°2¥- _wandbruntime‚1764  _runtime‚1764§K{TY¦Ó /loss/FRt1/detection_loss‚1.7760405996097968 1loss/SRqqt4/detection_loss‚1.6087995767593384 1loss/FBt2/assignment_loss‚0.42302755158016225 /loss/FRt3/detection_loss‚3.2191745849947133 /loss/FRt4/detection_loss‚2.1597551990013857 /loss/FBt1/detection_loss‚0.7668457709215586 _runtime‚1764.996248647 2loss/SRqqt4/assignment_loss‚1.7565034422459025 1loss/FBt1/assignment_loss‚0.17105579808250934 /loss/SRqqt1/detection_loss‚1.17971131185952 /loss/FBt2/detection_loss‚3.9546955227851868 .loss/FBt4/detection_loss‚2.311412950808352 _step‚194 1loss/SRqqt3/assignment_loss‚1.925932119290034 .loss/FRt2/detection_loss‚2.518199043071016 /loss/FBt3/detection_loss‚3.0512982666492463 .loss/FRt1/assignment_loss‚1.82336864563135 epoch‚1 trainer/global_step‚2399 .loss/FRt3/assignment_loss‚4.31343212723732 1loss/FBt4/assignment_loss‚0.36337864149192517 1loss/SRqqt2/detection_loss‚2.0272689557501247 %loss/total_loss‚49.27867239358349 % _timestamp‚1.7884300802915802e+09 /loss/FRt2/assignment_loss‚3.977524270402624 2loss/SRqqt1/assignment_loss‚1.4050816912530808 2loss/SRqqt2/assignment_loss‚1.8516328853688069 0loss/FRt4/assignment_loss‚3.4090382777727566 1loss/FBt3/assignment_loss‚0.44877263605594564 1loss/SRqqt3/detection_loss‚2.8367225249608357°¡ï4^§Ø /loss/FRt3/detection_loss‚3.2191745849947133 /loss/FRt4/detection_loss‚2.1597551990013857 /loss/FBt1/detection_loss‚0.7668457709215586 _runtime‚1764.996248647 2loss/SRqqt4/assignment_loss‚1.7565034422459025 1loss/FBt1/assignment_loss‚0.17105579808250934 /loss/SRqqt1/detection_loss‚1.17971131185952 /loss/FBt2/detection_loss‚3.9546955227851868 .loss/FBt4/detection_loss‚2.311412950808352 _step‚194 1loss/SRqqt3/assignment_loss‚1.925932119290034 .loss/FRt2/detection_loss‚2.518199043071016 /loss/FBt3/detection_loss‚3.0512982666492463 .loss/FRt1/assignment_loss‚1.82336864563135 epoch‚1 trainer/global_step‚2399 .loss/FRt3/assignment_loss‚4.31343212723732 1loss/FBt4/assignment_loss‚0.36337864149192517 1loss/SRqqt2/detection_loss‚2.0272689557501247 %loss/total_loss‚49.27867239358349 % _timestamp‚1.7884300802915802e+09 /loss/FRt2/assignment_loss‚3.977524270402624 2loss/SRqqt1/assignment_loss‚1.4050816912530808 2loss/SRqqt2/assignment_loss‚1.8516328853688069 0loss/FRt4/assignment_loss‚3.4090382777727566 1loss/FBt3/assignment_loss‚0.44877263605594564 1loss/SRqqt3/detection_loss‚2.8367225249608357 /loss/FRt1/detection_loss‚1.7760405996097968 1loss/SRqqt4/detection_loss‚1.6087995767593384 1loss/FBt2/assignment_loss‚0.42302755158016225³˜“¨‚(:ˆ …ŽåÔˆÎý gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.0442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04´´ƒ#©‚(:˜ …ŽåÔàš‘ disk.dm-0.in‚0 disk.rbd1.out‚ 0.919921875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2862.53515625 disk.sda1.in‚0% proc.memory.availableMB‚ 5.3046875 cpu‚25.2116594743087 disk.dm-0.out‚9163.845703125 disk.dm-1.out‚ 73.27734375 disk.sda2.in‚0 disk.rbd2.out‚72.14794921875 proc.cpu.threads‚17% disk./.usageGB‚1099.7522239685059 disk.drbd1777.in‚0 disk.rbd4.out‚0 disk.rbd1.in‚0% disk.nvme0c0n1.out‚9163.845703125 disk.rbd0.in‚0 network.sent‚ 4.331561e+06) disk./.usagePercent‚73.80529450410398 disk.rbd4.in‚0 disk.sda.out‚ 73.27734375 disk.nvme0n1.in‚0$ memory_percent‚99.96762275695801) proc.memory.percent‚17.47152805328369 network.recv‚683219 disk.sda3.out‚ 73.27734375 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚9163.845703125 disk.nvme0c0n1.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0ÉŒ™ñžª‚(:“ ”ŽåÔ‚ŠÛ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚162.784' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.556799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Tr>#«‚(:˜ ”ŽåÔ÷è• disk.sda3.in‚0 disk.sda3.out‚ 73.7109375$ proc.memory.availableMB‚3.984375) proc.memory.percent‚17.63608455657959% disk./.usageGB‚1099.7521896362305 disk.sda.in‚0 disk.dm-1.out‚ 73.7109375 disk.sda2.out‚0 disk.rbd2.out‚ 73.74609375 proc.cpu.threads‚17 network.sent‚ 4.441384e+06) disk./.usagePercent‚73.80529220003618 disk.sda2.in‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 0.919921875 disk.rbd2.in‚0$ memory_percent‚99.97568130493164# proc.memory.rssMB‚ 2889.49609375 disk.rbd4.out‚0 disk.sda.out‚ 73.7109375 disk.rbd1.in‚0 disk.drbd1777.out‚0 network.recv‚688953 disk.sda1.in‚0 cpu‚25.22923633335175 disk.rbd3.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚9263.02294921875 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚9263.02294921875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚0" disk.dm-0.out‚9263.02294921875 disk.dm-1.in‚0 disk.nvme0n1.in‚0Åo”¬‚(:’ £ŽåÔ¾…¸ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚158.288' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.657600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bú;(­‚(: £ŽåÔ€ËÔ disk.rbd0.in‚0* proc.memory.percent‚17.076945304870605 network.recv‚693552 disk.dm-0.in‚0 disk.sda.out‚ 74.5078125 disk.sda3.in‚0) disk./.usagePercent‚73.80528835992315 disk.sda1.out‚0 disk.rbd1.out‚ 0.919921875 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 73.76171875% disk./.usageGB‚1099.7521324157715 disk.sda.in‚0 disk.sda3.out‚ 74.5078125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2797.88671875 cpu‚25.268712741600982" disk.dm-0.out‚9362.06982421875 disk.nvme0c0n1.in‚0$ memory_percent‚99.40345287322998' proc.memory.availableMB‚ 97.73828125 proc.cpu.threads‚17 disk.dm-1.out‚ 74.5078125 disk.nvme0n1.in‚0 network.sent‚ 4.445459e+06% disk.nvme0n1.out‚9362.06982421875' disk.nvme0c0n1.out‚9362.06982421875«|’®‚(:‡ ²ŽåÔœÓÔ gpu.0.gpu‚58.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚86.759' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.3518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿¯ ¯‚(:• ²Žå԰ᔑ disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 74.82421875 disk.drbd1777.out‚0 network.sent‚ 4.450806e+06" disk.nvme0n1.out‚ 9461.44140625 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 9461.44140625 disk.rbd3.out‚0 disk.sda.in‚0' proc.memory.availableMB‚ 33.30078125$ disk./.usageGB‚1099.751392364502 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚0" proc.memory.rssMB‚ 2864.0859375 proc.cpu.threads‚17) disk./.usagePercent‚73.80523869446152 disk.rbd1.out‚ 0.919921875 disk.nvme0c0n1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd2.out‚ 73.82421875 disk.dm-0.out‚ 9461.44140625 disk.sda.out‚ 74.82421875* proc.memory.percent‚17.480993270874023 disk.dm-1.out‚ 74.82421875 disk.rbd4.out‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd4.in‚0$ memory_percent‚99.79674816131592 cpu‚25.311321928918193 network.recv‚697159¦vAV2°- _wandbruntime‚1824  _runtime‚1824%k9ív±ºT NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚45619235 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚35122574 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚45635210 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15975 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚11642226 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863814656 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510832128 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2024221636096 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚150 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2016279805479 % _timestamp‚1.7884301402038782e+09 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24126 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2024221636096 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚45635210 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚155521323205120 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853513 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚140459876684800 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚11642078 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 _step‚195 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 137187840 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚140459858681856 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15977 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚20492542 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚32134768 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510832128 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955585024 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914384 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚80757784 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚32134618 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚80757784 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚142483587488768 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚153395871873024 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492829184 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699017 _runtime‚1824.909037847 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚142484098320896 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492829184 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚142484098320896 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚140459858681856 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚148 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚155521460392960 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚142483587488768 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚35122574 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚35122572 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 trainer/global_step‚2417dƒ´E*²¿T YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚153395871873024 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492829184 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699017 _runtime‚1824.909037847 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚142484098320896 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492829184 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚142484098320896 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚140459858681856 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚148 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚155521460392960 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚142483587488768 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚35122574 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚35122572 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 trainer/global_step‚2417 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚45619235 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚35122574 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚45635210 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15975 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚11642226 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863814656 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510832128 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2024221636096 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚150 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2016279805479 % _timestamp‚1.7884301402038782e+09 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24126 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2024221636096 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚45635210 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚155521323205120 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853513 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚140459876684800 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚11642078 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 _step‚195 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 137187840 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚140459858681856 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15977 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚20492542 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚32134768 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510832128 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955585024 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914384 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚80757784 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚32134618 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚80757784 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚142483587488768à ¸2³- _wandbruntime‚1825  _runtime‚1825ó(¬oJ)´ÄR JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 trainer/global_step‚2417 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚35133378 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2024093933568 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955585024 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚148 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 113247744 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2017134329271 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24126 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15977 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚32145764 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚142521293476352 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚45626537 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚20499046 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955585024 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2125840398848 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚136816987074560 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24126 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚150 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚45642512 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚140496706713600 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚80759915 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24399 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492829184 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚80759915 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚142521293476352 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510832128 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚32145914 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚155559701253120 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚140496688710656 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699017 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 _runtime‚1825.796371271 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2024586762752 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚45642512 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚35133380 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚140496688710656 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863814656 % _timestamp‚1.7884301410912395e+09 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚11646720 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚153433747606528 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853513 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚140496706713600 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚142520782644224 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15975 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863814656 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚138834121403831 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚138833613450087 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚136816970035200 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2024093933568 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚20499044 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚80775892 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚45626537 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15977 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚35133378 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 137187840 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚35133380 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2125953646592 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚155559564065280 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914384 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚196 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚142520782644224 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829184 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚80775892 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15975 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2024586762752 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953744 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚41455ÔRµÉR G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15977 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚35133378 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 137187840 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚35133380 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2125953646592 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚155559564065280 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914384 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚196 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚142520782644224 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829184 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚80775892 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15975 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2024586762752 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953744 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 trainer/global_step‚2417 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚35133378 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2024093933568 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955585024 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚148 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 113247744 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2017134329271 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24126 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15977 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚32145764 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚142521293476352 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚45626537 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚20499046 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955585024 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2125840398848 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚136816987074560 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24126 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚150 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚45642512 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚140496706713600 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚80759915 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24399 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492829184 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚80759915 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚142521293476352 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510832128 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚32145914 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚155559701253120 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚140496688710656 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699017 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 _runtime‚1825.796371271 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2024586762752 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚45642512 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚35133380 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚140496688710656 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863814656 % _timestamp‚1.7884301410912395e+09 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚11646720 hâÖý TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚153433747606528 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853513 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚140496706713600 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚142520782644224 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15975 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863814656 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚138834121403831 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚138833613450087 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚136816970035200 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2024093933568 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚20499044 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚80775892 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚45626537 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844ÄZÙb“¶‚(:ˆ ÁŽåÔž¬¥Ž gpu.0.gpu‚49.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚159.829' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.9658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ν;µ·‚(:‰ ÁŽåÔ‚ùÅ disk.rbd4.in‚0 disk.dm-0.in‚0 disk.dm-1.out‚75.5625 disk.nvme0n1.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.480993270874023 cpu‚25.35138804670491 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd2.out‚ 73.83984375 network.sent‚ 4.456996e+06% disk./.usageGB‚1099.7513580322266 disk.sda.in‚0 disk.sda.out‚75.5625 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 9560.48828125 disk.rbd0.in‚0 disk.dm-0.out‚ 9560.48828125 disk.dm-1.in‚0" disk.nvme0n1.out‚ 9560.48828125 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.sda3.out‚75.5625 disk.drbd1777.in‚0" proc.memory.rssMB‚ 2864.0859375 disk.rbd3.out‚0 disk.rbd1.out‚ 0.919921875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0$ memory_percent‚99.79159832000732 network.recv‚701171) disk./.usagePercent‚73.80523639039372 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0' proc.memory.availableMB‚ 34.14453125§Øç«2¸- _wandbruntime‚1838  _runtime‚1838ª£æl¹Ú¦ 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05194985494017601 particle/accuracy_1_of_3‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2604166567325592 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0022675737272948027 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1577380895614624 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.15395523607730865 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3963938355445862 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.16161614656448364 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.33075353503227234 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0699993297457695 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11437133699655533 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18936780095100403 ;"Purity/2FRt1SRqqt2FBt/event_purity‚0.022727273404598236 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.442084938287735 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.055555559694767 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4479784369468689 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11401569843292236 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3434637188911438 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3894495368003845 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.48860788345336914 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5485265254974365 ,jet/accuracy_3_of_3‚0.014988799579441547 +particle/specificity‚0.9441694021224976 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16632263362407684 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.42092958092689514 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1380249559879303 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5235633850097656 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.29330354928970337 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4541271924972534 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4611697494983673 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.514285683631897 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4703651964664459 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39716774225234985 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01165712345391512 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.34205424785614014 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.364309698343277 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 8 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.02083333395421505 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12366301566362381 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319237709045 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.019489247351884842 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5123456716537476 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.29411765933036804 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3644971251487732 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12728406488895416 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4726221561431885 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27718591690063477 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00133200129494071 particle/accuracy_3_of_5‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926493167877197 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_5_of_5‚0.001792114693671465 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3437500298023224 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 particle/accuracy_4_of_7‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 trainer/global_step‚2417 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5080397725105286 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10924980044364929 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 _step‚197 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9820910692214966 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.36715832352638245 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_1‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19408364593982697 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.11847352981567383 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_2_of_6‚1 jet/accuracy_2_of_7‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.966749906539917 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13984675705432892 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 % _timestamp‚1.7884301532938101e+09 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.44607987999916077 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12144672125577927 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989227056503296 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.374474436044693 particle/accuracy_3_of_4‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07037711888551712 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 *jet/accuracy_1_of_5‚0.9505441188812256 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 epoch‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.45779645442962646 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.44817763566970825 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4390733540058136 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3055555522441864 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.15653152763843536 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.037236813455820084 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01596350409090519 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.07973423600196838 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20234858989715576 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 (particle/accuracy‚0.8345165848731995 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4823087453842163 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17668136954307556 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 (particle/f_score‚0.31306618452072144 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5530402660369873 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9647988080978394 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49398109316825867 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_6_of_6‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13027021288871765 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1248813271522522 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009201010689139366 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4851316213607788 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_4‚0.30738887190818787 ,jet/accuracy_4_of_4‚0.002213064581155777 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10505112260580063 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9298245310783386 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47999998927116394 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 _runtime‚1838.000446206 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01956157200038433 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.29491543769836426 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1662982553243637 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3915669620037079 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.060219891369342804 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975377321243286 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004913076292723417 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44902825355529785 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5535714030265808 +particle/sensitivity‚0.2417595535516739 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12040186673402786 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.548122763633728 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2796296775341034 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9820126295089722 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9914625883102417 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5387713313102722 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26292237639427185 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.36692020297050476 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982864260673523 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5031601786613464 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5479822754859924 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.45485639572143555 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6425964832305908 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14404642581939697 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.566319465637207 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606062412261963 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11522633582353592 *jet/accuracy_1_of_3‚0.8386744260787964 particle/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09648556262254715 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6222355961799622 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_2_of_4‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17402522265911102 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9898489117622375 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.09615384787321091 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2867458164691925 particle/accuracy_2_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.32692307233810425 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12246968597173691 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2593854069709778 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1957811564207077 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16158990561962128 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.04820788651704788 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9869058728218079 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9897514581680298 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1243462786078453 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4123326241970062 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5114942193031311 *jet/accuracy_1_of_4‚0.8919212222099304 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.39939165115356445 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_5_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_5_of_6‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.04964539408683777 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.019704433158040047 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3499999940395355 jet/accuracy_7_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3123456835746765 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4466954171657562 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14261990785598755 jet/accuracy_1_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12894736230373383 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0046568321995437145 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19668729603290558 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9930741190910339 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11437133699655533 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.507798433303833 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4255317151546478 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4633035957813263 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2986111044883728 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2202938199043274 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.45086804032325745 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3768199384212494 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4027160406112671 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9812376499176025 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.011767333373427391 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5575397610664368 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9879939556121826 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9837726950645447 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17901235818862915 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982864260673523 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4903854429721832 +jet/accuracy_2_of_2‚0.07877910882234573 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 jet/accuracy_3_of_7‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9795371890068054 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1182447001338005 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5138888955116272 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3714297115802765 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4611697494983673 *jet/accuracy_1_of_2‚0.7197914719581604 jet/accuracy_4_of_7‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2412770688533783 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1578475832939148 6validation_average_jet_accuracy‚0.4051932096481323 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.19814816117286682 @&Purity/1FRt0SRqqt4FBt/event_propÄØüZ'ortion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3957778811454773 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9855693578720093 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.510219931602478 *jet/accuracy_2_of_3‚0.1932467371225357 +jet/accuracy_3_of_4‚0.03790387138724327 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.45661479234695435 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.35197368264198303 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 jet/accuracy_6_of_7‚0 particle/accuracy_3_of_7‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9931973218917847 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09741736948490143 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11081162840127945 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9907004833221436 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.015476190485060215 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.29629629850387573 particle/accuracy_3_of_6‚1 jet/accuracy_1_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.49907946586608887 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 jet/accuracy_5_of_7‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 )loss/val_total_loss‚50.57059525688377 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1183730736374855 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.004480286967009306 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.988540768623352 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18015365302562714 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43675243854522705 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9913668036460876 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.006122448947280645 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.28125 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02079501748085022 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9824773669242859 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.33472496271133423 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13054092228412628 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15172414481639862 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9953242540359497 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11364974081516266 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0037735849618911743 +jet/accuracy_2_of_5‚0.37625446915626526 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.532532811164856 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04561380669474602 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5259707570075989 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.040697675198316574 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3012649118900299 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30173608660697937 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12666666507720947 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4138948321342468 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16326062381267548 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6052631735801697 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47548794746398926 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3959694802761078 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4882628917694092 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5059207081794739 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29315367341041565 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47596457600593567 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.31578946113586426 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.27177897095680237 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.818836510181427 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19043532013893127ßV$C˜Xºߦ 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.27177897095680237 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.818836510181427 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19043532013893127 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05194985494017601 particle/accuracy_1_of_3‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2604166567325592 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0022675737272948027 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1577380895614624 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.15395523607730865 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3963938355445862 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.16161614656448364 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.33075353503227234 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0699993297457695 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11437133699655533 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18936780095100403 ;"Purity/2FRt1SRqqt2FBt/event_purity‚0.022727273404598236 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.442084938287735 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.055555559694767 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4479784369468689 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11401569843292236 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3434637188911438 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3894495368003845 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.48860788345336914 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5485265254974365 ,jet/accuracy_3_of_3‚0.014988799579441547 +particle/specificity‚0.9441694021224976 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16632263362407684 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.42092958092689514 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1380249559879303 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5235633850097656 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.29330354928970337 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4541271924972534 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4611697494983673 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.514285683631897 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4703651964664459 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39716774225234985 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01165712345391512 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.34205424785614014 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.364309698343277 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 8 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.02083333395421505 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12366301566362381 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319237709045 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.019489247351884842 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5123456716537476 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.29411765933036804 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3644971251487732 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12728406488895416 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4726221561431885 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27718591690063477 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00133200129494071 particle/accuracy_3_of_5‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926493167877197 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_5_of_5‚0.001792114693671465 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3437500298023224 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 particle/accuracy_4_of_7‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 trainer/global_step‚2417 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5080397725105286 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10924980044364929 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 _step‚197 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9820910692214966 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.36715832352638245 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_1‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19408364593982697 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.11847352981567383 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_2_of_6‚1 jet/accuracy_2_of_7‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.966749906539917 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13984675705432892 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 % _timestamp‚1.7884301532938101e+09 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.44607987999916077 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12144672125577927 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989227056503296 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.374474436044693 particle/accuracy_3_of_4‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07037711888551712 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 *jet/accuracy_1_of_5‚0.9505441188812256 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 epoch‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.45779645442962646 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.44817763566970825 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4390733540058136 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3055555522441864 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.15653152763843536 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.037236813455820084 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01596350409090519 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.07973423600196838 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20234858989715576 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 (particle/accuracy‚0.8345165848731995 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4823087453842163 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17668136954307556 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 (particle/f_score‚0.31306618452072144 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5530402660369873 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9647988080978394 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49398109316825867 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_6_of_6‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13027021288871765 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1248813271522522 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009201010689139366 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4851316213607788 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_4‚0.30738887190818787 ,jet/accuracy_4_of_4‚0.002213064581155777 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10505112260580063 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9298245310783386 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47999998927116394 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 _runtime‚1838.000446206 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01956157200038433 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.29491543769836426 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1662982553243637 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3915669620037079 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.060219891369342804 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975377321243286 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004913076292723417 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44902825355529785 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5535714030265808 +particle/sensitivity‚0.2417595535516739 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12040186673402786 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.548122763633728 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2796296775341034 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9820126295089722 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9914625883102417 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5387713313102722 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26292237639427185 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.36692020297050476 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982864260673523 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5031601786613464 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5479822754859924 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.45485639572143555 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6425964832305908 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14404642581939697 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.566319465637207 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606062412261963 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11522633582353592 *jet/accuracy_1_of_3‚0.8386744260787964 particle/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09648556262254715 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6222355961799622 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_2_of_4‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17402522265911102 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9898489117622375 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.09615384787321091 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2867458164691925 particle/accuracy_2_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.32692307233810425 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12246968597173691 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2593854069709778 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1957811564207077 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16158990561962128 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.04820788651704788 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9869058728218079 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9897514581680298 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1243462786078453 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4123326241970062 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5114942193031311 *jet/accuracy_1_of_4‚0.8919212222099304 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.39939165115356445 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_5_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.298611 á ÝÎ:1044883728 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_5_of_6‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.04964539408683777 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.019704433158040047 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3499999940395355 jet/accuracy_7_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3123456835746765 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4466954171657562 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14261990785598755 jet/accuracy_1_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12894736230373383 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0046568321995437145 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19668729603290558 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9930741190910339 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11437133699655533 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.507798433303833 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4255317151546478 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4633035957813263 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2986111044883728 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2202938199043274 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.45086804032325745 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3768199384212494 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4027160406112671 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9812376499176025 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.011767333373427391 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5575397610664368 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9879939556121826 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9837726950645447 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17901235818862915 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982864260673523 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4903854429721832 +jet/accuracy_2_of_2‚0.07877910882234573 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 jet/accuracy_3_of_7‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9795371890068054 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1182447001338005 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5138888955116272 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3714297115802765 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4611697494983673 *jet/accuracy_1_of_2‚0.7197914719581604 jet/accuracy_4_of_7‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2412770688533783 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1578475832939148 6validation_average_jet_accuracy‚0.4051932096481323 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.19814816117286682 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3957778811454773 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9855693578720093 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.510219931602478 *jet/accuracy_2_of_3‚0.1932467371225357 +jet/accuracy_3_of_4‚0.03790387138724327 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.45661479234695435 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.35197368264198303 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 jet/accuracy_6_of_7‚0 particle/accuracy_3_of_7‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9931973218917847 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09741736948490143 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11081162840127945 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9907004833221436 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.015476190485060215 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.29629629850387573 particle/accuracy_3_of_6‚1 jet/accuracy_1_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.49907946586608887 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 jet/accuracy_5_of_7‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 )loss/val_total_loss‚50.57059525688377 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1183730736374855 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.004480286967009306 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.988540768623352 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18015365302562714 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43675243854522705 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9913668036460876 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.006122448947280645 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.28125 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02079501748085022 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9824773669242859 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.33472496271133423 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13054092228412628 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15172414481639862 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9953242540359497 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11364974081516266 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0037735849618911743 +jet/accuracy_2_of_5‚0.37625446915626526 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.532532811164856 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04561380669474602 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5259707570075989 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.040697675198316574 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3012649118900299 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30173608660697937 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12666666507720947 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4138948321342468 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16326062381267548 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6052631735801697 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47548794746398926 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3959694802761078 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4882628917694092 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5059207081794739 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29315367341041565 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47596457600593567 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.31578946113586426 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75Ån%¯ž»‚(:“ ÐŽåÔòÎô gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚149.545' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.908999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹¦"+¼‚(:  ÐŽåÔ¸¿¥‘% disk./.usageGB‚1100.2151527404785( disk.nvme0c0n1.out‚10034.25634765625 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 network.recv‚706695 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 disk.rbd2.out‚73.99267578125 disk.sda3.out‚ 76.0234375 disk.sda1.in‚0( proc.memory.availableMB‚ 952.73046875 cpu‚25.3664545535131) disk./.usagePercent‚73.83636204241418 disk.dm-1.in‚0" proc.memory.rssMB‚ 2870.1953125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.out‚0# memory_percent‚94.1849946975708 disk.rbd0.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚10034.25634765625 disk.dm-1.out‚ 76.0234375 disk.sda.out‚ 76.0234375 network.sent‚ 4.563757e+06& disk.nvme0n1.out‚10034.25634765625 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.nvme0n1.in‚0 disk.dm-0.in‚0* proc.memory.percent‚17.518281936645508ùŒüȽ‚(:’ ߎåÔ랊Ž gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚89.821' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.964199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{º/–)¾‚(:ž ߎåÔ§ÄÁ‘# disk.dm-0.out‚10138.70751953125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚10138.70751953125 disk.sda2.in‚0 disk.rbd2.out‚75.52001953125 network.sent‚ 4.688038e+06( disk./.usagePercent‚73.8367061165402 disk.rbd4.out‚0 disk.sda.in‚0 disk.sda.out‚ 76.640625& disk.nvme0n1.out‚10138.70751953125 disk.rbd1.out‚ 0.919921875 disk.sda3.in‚0 network.recv‚712907 disk.nvme0n1.in‚0 disk.rbd0.out‚0$ memory_percent‚94.18425559997559 disk.dm-1.out‚ 76.640625 proc.cpu.threads‚17 disk.drbd1777.out‚0% disk./.usageGB‚1100.2202796936035 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0* proc.memory.percent‚17.518281936645508 disk.rbd3.in‚0 disk.dm-0.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 76.640625' proc.memory.availableMB‚ 952.8515625" proc.memory.rssMB‚ 2870.1953125 cpu‚25.363932102594166 disk.rbd4.in‚0üY›2¿- _wandbruntime‚1864  _runtime‚1864SY ÈÀ ' lr-AdamW/pg1‚0.00021943816159010346 ' lr-AdamW/pg2‚0.00021943816159010346 trainer/global_step‚2449 $ _timestamp‚1.788430179503599e+09 _runtime‚1864.208397958 _step‚198$¼B³ÍÁÇ _runtime‚1864.208397958 _step‚198 ' lr-AdamW/pg1‚0.00021943816159010346 ' lr-AdamW/pg2‚0.00021943816159010346 trainer/global_step‚2449 $ _timestamp‚1.788430179503599e+09ÆöȘ2Â- _wandbruntime‚1864  _runtime‚1864`šª´Ù'ÃÓO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46314927 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46314927 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚426 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138977816955904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚141024345495413 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144776516800000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2054093510656 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35708354 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35708354 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35708352 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2054093510656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157994729879552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20836806 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2054749872640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138977833995264 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144776516800000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46294890 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11845200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142722441292288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2157254953984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2047182977261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46294890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11844776 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2046528539509 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2046528539509 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144776516800000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2054093510656 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35708354 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35708354 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35708352 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2054093510656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157994729879552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Çt7¢‚2Å- _wandbruntime‚1864  _runtime‚1864ñ“.á&ÆÛM P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46333407 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142778776328192 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82055777 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 trainer/global_step‚2449 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46333407 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2055569313792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82035712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 # _timestamp‚1.78843018016783e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155898955747328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46313344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141080066060937 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35722368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2054912938496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158057232728064 _runtime‚1864.873050988 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11849668 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82035712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35722370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82055777 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141080737538257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35722370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46313344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2054912938496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142778794331136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2047999412433 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2047344974473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144834363644928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144833689266688 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32694211 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2158107078144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142778776328192 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11849668 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82035712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35722370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82055777 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141080737538257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35722370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46313344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2054912938496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142778794331136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2047999412433 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2047344974473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144834363644928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144833689266688 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32694211 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2158107078144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142778776328192 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46333407 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142778776328192 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82055777 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 trainer/global_step‚2449 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46333407 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2055569313792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82035712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 # _timestamp‚1.78843018016783e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155898955747328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46313344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141080066060937 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35722368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2054912938496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158057232728064 _runtime‚1864.873050988 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143810387362821 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147635154548224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33316192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚2499 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21245206 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚203 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147635154548224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141723040523264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2095735678464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2095079316480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2200298192896 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36409154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33315736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚158919817952768 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2095079316480 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12070532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158919841892864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2200132471296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83650133 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36409154 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884302150384681e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145540093234688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161119950424064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928%=©Þ'ÔØO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47240979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47220942 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36409152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145540093234688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21245204 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36409152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2087363878917 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47220942 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161120140085760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143810387362821 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147635154548224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33316192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚2499 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21245206 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚203 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147635154548224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141723040523264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2095735678464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2095079316480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2200298192896 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36409154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33315736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚158919817952768 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2095079316480 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12070532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158919841892864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2200132471296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83650133 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36409154 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884302150384681e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145540093234688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161119950424064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚147635828913152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12070986 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889Ëþhå2Õ- _wandbruntime‚1900  _runtime‚19003ª’•ã&ÖÝM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83662564 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158981492630528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21253374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83682629 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36423170 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2096555119616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83682629 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36423168 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 trainer/global_step‚2499 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47259459 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33328496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚147693001393152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2096555119616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145596428270592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12075124 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2088180313881 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2200984711168 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2095898744320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36423170 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47239396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12075582 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2201154613760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145596446273536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2095898744320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141777927614464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884302156987333e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36423168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161182453401600 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143866107928345 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141777944653824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33328956 _runtime‚1900.404025718 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83662564 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47259459 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145596428270592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12075124 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2088180313881 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2200984711168 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2095898744320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36423170 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47239396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12075582 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2201154613760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145596446273536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2095898744320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141777927614464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884302156987333e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36423168 L>DeviceStatsMonitor.on_train_batch_/±8^ëend/active_bytes.all.current‚ 674378240 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161182453401600 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143866107928345 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141777944653824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33328956 _runtime‚1900.404025718 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83662564 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47259459 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83682629 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36423170 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2096555119616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83682629 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36423168 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 trainer/global_step‚2499 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47259459 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33328496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚147693001393152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2096555119616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872Ì|œ\›2Ø- _wandbruntime‚1900  _runtime‚1900jF•ZXÙÒ 0loss/FRt1/assignment_loss‚1.6225637243344233 /loss/FBt1/detection_loss‚0.6084661542984823 epoch‚2 trainer/global_step‚2499 2loss/SRqqt1/assignment_loss‚0.9046143328564035 2loss/SRqqt3/assignment_loss‚2.6661005154252053 /loss/FRt3/detection_loss‚3.5259756531034196 /loss/FBt4/detection_loss‚2.2550021960185123 0loss/FRt4/assignment_loss‚3.3927345154236774 0loss/FBt1/assignment_loss‚0.2191886504585507 1loss/SRqqt1/detection_loss‚1.5979064780252952 %loss/total_loss‚52.21429002133368 1loss/SRqqt2/assignment_loss‚1.651691643951031 -loss/FRt1/detection_loss‚2.10962265672592 /loss/FRt2/detection_loss‚2.3408402942798356 /loss/FRt4/detection_loss‚2.0971337581167417 1loss/SRqqt4/assignment_loss‚1.789588768373836 0loss/FBt4/assignment_loss‚0.7079592117896447 1loss/SRqqt2/detection_loss‚1.8161269924961603 1loss/SRqqt4/detection_loss‚1.7304600991986014 0loss/FBt2/assignment_loss‚1.0136694572865963 .loss/FBt3/detection_loss‚3.913391249520438 _step‚205 0loss/FRt2/assignment_loss‚3.4904411760243503 0loss/FBt3/assignment_loss‚0.6049804863120828 1loss/SRqqt3/detection_loss‚2.3800223872065542 .loss/FBt2/detection_loss‚5.040217697620392 /loss/FRt3/assignment_loss‚4.735591922487531 % _timestamp‚1.7884302157005394e+09 _runtime‚1900.405566882 u 5]Ú× /loss/FRt3/assignment_loss‚4.735591922487531 % _timestamp‚1.7884302157005394e+09 _runtime‚1900.405566882 0loss/FRt1/assignment_loss‚1.6225637243344233 /loss/FBt1/detection_loss‚0.6084661542984823 epoch‚2 trainer/global_step‚2499 2loss/SRqqt1/assignment_loss‚0.9046143328564035 2loss/SRqqt3/assignment_loss‚2.6661005154252053 /loss/FRt3/detection_loss‚3.5259756531034196 /loss/FBt4/detection_loss‚2.2550021960185123 0loss/FRt4/assignment_loss‚3.3927345154236774 0loss/FBt1/assignment_loss‚0.2191886504585507 1loss/SRqqt1/detection_loss‚1.5979064780252952 %loss/total_loss‚52.21429002133368 1loss/SRqqt2/assignment_loss‚1.651691643951031 -loss/FRt1/detection_loss‚2.10962265672592 /loss/FRt2/detection_loss‚2.3408402942798356 /loss/FRt4/detection_loss‚2.0971337581167417 1loss/SRqqt4/assignment_loss‚1.789588768373836 0loss/FBt4/assignment_loss‚0.7079592117896447 1loss/SRqqt2/detection_loss‚1.8161269924961603 1loss/SRqqt4/detection_loss‚1.7304600991986014 0loss/FBt2/assignment_loss‚1.0136694572865963 .loss/FBt3/detection_loss‚3.913391249520438 _step‚205 0loss/FRt2/assignment_loss‚3.4904411760243503 0loss/FBt3/assignment_loss‚0.6049804863120828 1loss/SRqqt3/detection_loss‚2.3800223872065542 .loss/FBt2/detection_loss‚5.040217697620392ÍÌ¿^ëžÛ‚(:“ ŒåÔŽçþ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚143.306' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.661200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïå0)Ü‚(:ž ŒåÔȚđ disk.rbd1.out‚ 0.919921875 disk.sda3.out‚ 78.578125 disk.drbd1777.in‚0* proc.memory.percent‚17.518281936645508 cpu‚25.356615273876322) disk./.usagePercent‚73.83665184294294& disk.nvme0n1.out‚10911.27197265625 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd4.in‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚10911.27197265625 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0% disk./.usageGB‚1100.2194709777832 disk.rbd3.out‚0' proc.memory.availableMB‚ 951.7265625" proc.memory.rssMB‚ 2870.1953125 disk.rbd0.in‚0# disk.dm-0.out‚10911.27197265625 disk.dm-1.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0$ memory_percent‚94.19112205505371 disk.sda.in‚0 disk.sda.out‚ 78.578125 disk.sda1.in‚0 disk.dm-1.out‚ 78.578125 disk.sda3.in‚0 network.sent‚ 4.807979e+06 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0 disk.sda2.in‚0 disk.rbd2.out‚ 77.3603515625 network.recv‚727437 disk.sda1.out‚0 disk.nvme0n1.in‚02®›Ï“Ý‚(:ˆ ›åÔÖ…î gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚107.537' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5074 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùr û-Þ‚(:¢ ›åÔÎ˼$ memory_percent‚94.19245719909668% disk./.usageGB‚1100.2194519042969& disk.nvme0n1.out‚10911.61962890625* proc.memory.percent‚17.518281936645508 disk.rbd2.out‚ 77.3720703125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2870.1953125 disk.rbd2.in‚0 disk.dm-1.out‚ 79.30859375 network.sent‚ 4.9166e+06( disk.nvme0c0n1.out‚10911.61962890625 disk.sda3.out‚ 79.30859375 disk.drbd1777.in‚0# disk.dm-0.out‚10911.61962890625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.sda1.out‚0 cpu‚25.354408987296107 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.dm-0.in‚0 disk.nvme0n1.in‚0' proc.memory.availableMB‚ 951.5078125 network.recv‚733335) disk./.usagePercent‚73.83665056290528 disk.rbd1.out‚ 0.919921875 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda.out‚ 79.30859375 disk.sda1.in‚0=¿ƒnœß‚(:‘ ªåÔÆ ŽŽ gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.7' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.540000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/m,à‚(:¡ ªåÔàÌø" proc.memory.rssMB‚ 2870.1953125 disk.drbd1777.in‚0 disk.drbd1777.out‚0$ disk./.usageGB‚1100.219051361084( disk.nvme0c0n1.out‚11011.41455078125 disk.rbd2.out‚ 77.4345703125 disk.rbd3.in‚0 disk.sda1.in‚0* proc.memory.percent‚17.518281936645508 disk.rbd0.out‚0 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.sda2.in‚0 disk.rbd4.in‚0# disk.dm-0.out‚11011.41455078125 disk.sda.out‚ 79.9609375 disk.dm-1.out‚ 79.9609375 network.sent‚ 4.921957e+06 network.recv‚737209 disk.nvme0c0n1.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 cpu‚25.352067926912465) disk./.usagePercent‚73.83662368211418& disk.nvme0n1.out‚11011.41455078125 disk.sda2.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 951.47265625 proc.cpu.threads‚17 disk.rbd1.out‚ 0.919921875 disk.sda3.out‚ 79.9609375 disk.rbd1.in‚0 disk.sda3.in‚0 disk.rbd4.out‚0 disk.sda1.out‚0$ memory_percent‚94.19267177581787IÌ2á- _wandbruntime‚1935  _runtime‚1935¦"OÉâà ' lr-AdamW/pg1‚0.00021824768028336886 ' lr-AdamW/pg2‚0.00021824768028336886 trainer/global_step‚2549 % _timestamp‚1.7884302506290438e+09 _runtime‚1935.333766612 _step‚206Lä¦ÎãÈ _runtime‚1935.333766612 _step‚206 ' lr-AdamW/pg1‚0.00021824768028336886 ' lr-AdamW/pg2‚0.00021824768028336886 trainer/global_step‚2549 % _timestamp‚1.7884302506290438e+09ΛÐ*2ä- _wandbruntime‚1935  _runtime‚1935ðC÷Ù'åÓO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33949723 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚2549 _runtime‚1935.334812723 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚148357727174144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37109952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12296119 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚144468247051264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164245513569280 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2136721484288 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚148357727174144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144468230011904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 % _timestamp‚1.7884302506297565e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37109954 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21653604 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2136065122304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37109954 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150493792296448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162002378776064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48146994 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150493792296448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48167031 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148357745177088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48146994 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚146597100707341 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150494466661376 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85256946 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33950157 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12296551 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162002354835968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2128199218325 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2136065122304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2243316066304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150494466661376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273088 _step‚207 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0Ý 6­›æØO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37109952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12296119 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚144468247051264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164245513569280 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2136721484288 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚148357727174144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144468230011904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 % _timestamp‚1.7884302506297565e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37109954 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21653604 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2136065122304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37109954 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150493792296448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162002378776064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48146994 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150493792296448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48167031 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148357745177088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48146994 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚146597100707341 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150494466661376 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85256946 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33950157 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12296551 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1‚©´äC 62002354835968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2128199218325 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2136065122304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2243316066304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150494466661376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273088 _step‚207 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33949723 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚2549 _runtime‚1935.334812723 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚148357727174144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Ï麟Y2ç- _wandbruntime‚1935  _runtime‚1935ò ã&èÝM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21661772 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162064005573632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85289416 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2129670091249 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85309481 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85309481 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162064029513728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148414080212992 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85289416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2136884550144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚148414098215936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48165448 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2137540925440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33962776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164308213736960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37123970 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150551639141376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2136884550144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33962332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21661774 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2244022709248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48185511 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884302512917662e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37123968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164308028282880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513984 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150550964763136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2137540925440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144523151181824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2129015653289 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37123970 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2244184223232 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48185511 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48165448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12301002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146652821273073 trainer/global_step‚2549 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚148414080212992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12300560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37123968˜Gæcè&éâM FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85309481 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162064029513728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148414080212992 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85289416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2136884550144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚148414098215936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48165448 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2137540925440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33962776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164308213736960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37123970 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150551639141376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2136884550144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33962332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21661774 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2244022709248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48185511 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884302512917662e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37123968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164308028282880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513984 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150550964763136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2137540925440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144523151181824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2129015653289 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37123970 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2244184223232 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48185511 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48165448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12301002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146652821273073 trainer/global_step‚2549 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚148414080212992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12300560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37123968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21661772 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162064005573632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85289416 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2129670091249 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85309481 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30ÐmSÿ'2ê- _wandbruntime‚1935  _runtime‚1935o•WëÑ 1loss/SRqqt2/assignment_loss‚1.457553469731162 /loss/FRt4/detection_loss‚2.4283218770413786 /loss/FBt1/detection_loss‚0.8056820283395573 %loss/total_loss‚50.49501595711173 /loss/FRt3/assignment_loss‚4.785744121580413 .loss/FRt1/detection_loss‚1.989931830972956 2loss/SRqqt4/assignment_loss‚1.7694854182004929 0loss/FBt1/assignment_loss‚0.2476188543553507 1loss/SRqqt1/detection_loss‚1.1815148081098283 1loss/SRqqt2/detection_loss‚1.9964528207977612 1loss/SRqqt3/detection_loss‚2.4366496511868068 /loss/FBt3/detection_loss‚2.8286065220832826 _runtime‚1935.998456978 .loss/FRt1/assignment_loss‚2.16880660726313 /loss/FRt4/assignment_loss‚3.595335258019937 0loss/FBt3/assignment_loss‚0.5635481506585733 0loss/FBt4/assignment_loss‚0.6881578597757552 % _timestamp‚1.7884302512938225e+09 1loss/SRqqt4/detection_loss‚2.0949393522739412 1loss/FBt2/assignment_loss‚0.40818555614290136 /loss/FRt2/detection_loss‚2.3208693276752124 .loss/FBt2/detection_loss‚3.457686851421992 _step‚209 trainer/global_step‚2549 .loss/FRt3/detection_loss‚2.683048177849163 /loss/FBt4/detection_loss‚3.5428675611813865 epoch‚2 0loss/FRt2/assignment_loss‚3.6691844940185545 1loss/SRqqt1/assignment_loss‚0.975210929171197 2loss/SRqqt3/assignment_loss‚2.3996144292610033Ç'@\ìÖ .loss/FBt2/detection_loss‚3.457686851421992 _step‚209 trainer/global_step‚2549 .loss/FRt3/detection_loss‚2.683048177849163 /loss/FBt4/detection_loss‚3.5428675611813865 epoch‚2 0loss/FRt2/assignment_loss‚3.6691844940185545 1loss/SRqqt1/assignment_loss‚0.975210929171197 2loss/SRqqt3/assignment_loss‚2.3996144292610033 1loss/SRqqt2/assignment_loss‚1.457553469731162 /loss/FRt4/detection_loss‚2.4283218770413786 /loss/FBt1/detection_loss‚0.8056820283395573 %loss/total_loss‚50.49501595711173 /loss/FRt3/assignment_loss‚4.785744121580413 .loss/FRt1/detection_loss‚1.989931830972956 2loss/SRqqt4/assignment_loss‚1.7694854182004929 0loss/FBt1/assignment_loss‚0.2476188543553507 1loss/SRqqt1/detection_loss‚1.1815148081098283 1loss/SRqqt2/detection_loss‚1.9964528207977612 1loss/SRqqt3/detection_loss‚2.4366496511868068 /loss/FBt3/detection_loss‚2.8286065220832826 _runtime‚1935.998456978 .loss/FRt1/assignment_loss‚2.16880660726313 /loss/FRt4/assignment_loss‚3.595335258019937 0loss/FBt3/assignment_loss‚0.5635481506585733 0loss/FBt4/assignment_loss‚0.6881578597757552 % _timestamp‚1.7884302512938225e+09 1loss/SRqqt4/detection_loss‚2.0949393522739412 1loss/FBt2/assignment_loss‚0.40818555614290136 /loss/FRt2/detection_loss‚2.3208693276752124ÑPG6ží‚(:“ ¹åÔ‘š‰Ž gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.896' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.179199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xý‘Å(î‚(: ¹åÔÑæ network.recv‚741779 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd4.out‚0 disk.sda.out‚ 80.703125( disk./.usagePercent‚73.8366224020765 disk.rbd2.in‚0 disk.rbd2.out‚ 78.9619140625# disk.dm-0.out‚11110.50048828125( proc.memory.availableMB‚ 950.77734375 disk.dm-1.out‚ 80.703125 disk.sda3.in‚0 disk.sda3.out‚ 80.703125" proc.memory.rssMB‚ 2870.1953125 proc.cpu.threads‚17 network.sent‚ 4.928182e+06% disk./.usageGB‚1100.2190322875977 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚0 disk.sda.in‚0 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚11110.50048828125 disk.sda2.in‚0 disk.rbd3.out‚0$ memory_percent‚94.19691562652588* proc.memory.percent‚17.518281936645508& disk.nvme0n1.out‚11110.50048828125 disk.nvme0n1.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.005859375 disk.rbd4.in‚0 cpu‚25.34977862113057eŒÈŽžï‚(:“ ÈåÔ½Ÿ‰Ž gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.942' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.188399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I_¨dð‚(:” ÈåÔõÝý' proc.memory.availableMB‚ 950.4609375 disk.rbd3.out‚0 disk.nvme0c0n1.in‚0 disk.rbd4.out‚0 disk.dm-0.in‚0! disk.nvme0n1.out‚ 11209.921875 disk.rbd1.in‚0 disk.drbd1777.in‚0% disk./.usageGB‚1100.2190589904785 disk.dm-1.out‚ 81.11328125 disk.rbd1.out‚ 1.068359375 cpu‚25.347515255654244 network.recv‚747802 disk.rbd4.in‚0 disk.dm-0.out‚ 11209.921875 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.518281936645508 network.sent‚ 5.036143e+06 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0# memory_percent‚94.1988468170166" proc.memory.rssMB‚ 2870.1953125) disk./.usagePercent‚73.83662419412924 disk.sda.out‚ 81.11328125# disk.nvme0c0n1.out‚ 11209.921875 disk.rbd2.out‚ 79.0869140625 disk.sda3.out‚ 81.11328125UpKy2ñ- _wandbruntime‚1970  _runtime‚1970#«4Èò $ _timestamp‚1.788430286118109e+09 _runtime‚1970.822930465 _step‚210 ' lr-AdamW/pg1‚0.00021761957361620177 ' lr-AdamW/pg2‚0.00021761957361620177 trainer/global_step‚2599œuÍŠÍóÇ ' lr-AdamW/pg1‚0.00021761957361620177 ' lr-AdamW/pg2‚0.00021761957361620177 trainer/global_step‚2599 $ _timestamp‚1.788430286118109e+09 _runtime‚1970.822930465 _step‚210ÒÉéà¦2ô- _wandbruntime‚1970  _runtime‚1970å¿‚2ï õÓO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153352430044672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22062004 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147213436539904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚149382471097637 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37810752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2177707290112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147213453579264 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2286078540288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34584095 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49093083 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49073046 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153353104409600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151175379116544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37810752 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153352430044672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2169688995485 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12522091 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167370970259456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153353104409600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37810754 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165084891719168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884302861189103e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86883798 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165084915659264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚211 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚419 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚417 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2177050928128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49073046 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2177050928128 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86903837"•ì#Þ'öØO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34584095 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49093083 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49073046 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153353104409600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151175379116544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37810752 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153352430044672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2169688995485 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12522091 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167370970259456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153353104409600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37810754 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165084891719168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884302861189103e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86883798 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165084915659264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚211 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚419 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚417 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2177050928128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49073046 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2177050928128 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86903837 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153352430044672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22062004 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147213436539904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚149382471097637 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37810752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2177707290112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147213453579264 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2286078540288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944Óm•þ2÷- _wandbruntime‚1971  _runtime‚1971®d)ßã&øÝM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147268357709824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167433483933696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165146566396928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2286941476864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2169850992697 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37824768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22070174 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2178526731264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86936333 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34596711 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚418 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151231732155392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151231750158336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86936333 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2177870355968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49091500 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12526957 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49091500 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2178526731264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149438863140481 _runtime‚1971.481014448 _step‚212 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153410276889600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884302867757916e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165146542456832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147268340670464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151231750158336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37824770 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚2599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021Œè‘ùâM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2286941476864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2169850992697 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37824768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22070174 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2178526731264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86936333 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34596711 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚418 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151231732155392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151231750158336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86936333 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2177870355968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49091500 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12526957 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_Îÿµ èpool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49091500 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2178526731264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149438863140481 _runtime‚1971.481014448 _step‚212 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153410276889600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884302867757916e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165146542456832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147268340670464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151231750158336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37824770 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚2599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147268357709824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167433483933696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165146566396928ÔïaU€2ú- _wandbruntime‚1971  _runtime‚1971 ¶ŒÀ[ûÕ 0loss/FRt2/assignment_loss‚3.7220923058364703 /loss/FBt1/detection_loss‚0.9520712536531415 trainer/global_step‚2599 _runtime‚1971.482343282 0loss/FBt3/assignment_loss‚0.4872112415703896 .loss/FRt2/detection_loss‚2.470674549755843 /loss/FRt4/assignment_loss‚4.133459482644056 1loss/SRqqt3/assignment_loss‚2.307544690267793 1loss/FBt2/assignment_loss‚0.44776309188455343 1loss/SRqqt3/detection_loss‚2.5048105264532157 .loss/FBt2/detection_loss‚3.179199233651161 0loss/FRt1/assignment_loss‚1.9853053712377362 /loss/FRt3/assignment_loss‚4.002987885475159 1loss/SRqqt4/detection_loss‚1.8867454522965001 .loss/FBt4/detection_loss‚2.532242293541248 % _timestamp‚1.7884302867777026e+09 1loss/FBt4/assignment_loss‚0.37050225764799577 _step‚213 2loss/SRqqt2/assignment_loss‚1.5457088649272919 1loss/FBt1/assignment_loss‚0.19299091353536973 /loss/FRt1/detection_loss‚1.9403091747971142 /loss/FRt4/detection_loss‚2.7249213561818406 1loss/SRqqt2/detection_loss‚1.9651318697296842 epoch‚2 2loss/SRqqt4/assignment_loss‚1.8118993035134148 .loss/FRt3/detection_loss‚2.969649772842725 /loss/FBt3/detection_loss‚2.8150538831949232 %loss/total_loss‚48.76905834882652 2loss/SRqqt1/assignment_loss‚0.9335726131958997 1loss/SRqqt1/detection_loss‚0.8872109609929955Ä2fã`üÚ 1loss/FBt4/assignment_loss‚0.37050225764799577 _step‚213 2loss/SRqqt2/assignment_loss‚1.5457088649272919 1loss/FBt1/assignment_loss‚0.19299091353536973 /loss/FRt1/detection_loss‚1.9403091747971142 /loss/FRt4/detection_loss‚2.7249213561818406 1loss/SRqqt2/detection_loss‚1.9651318697296842 epoch‚2 2loss/SRqqt4/assignment_loss‚1.8118993035134148 .loss/FRt3/detection_loss‚2.969649772842725 /loss/FBt3/detection_loss‚2.8150538831949232 %loss/total_loss‚48.76905834882652 2loss/SRqqt1/assignment_loss‚0.9335726131958997 1loss/SRqqt1/detection_loss‚0.8872109609929955 0loss/FRt2/assignment_loss‚3.7220923058364703 /loss/FBt1/detection_loss‚0.9520712536531415 trainer/global_step‚2599 _runtime‚1971.482343282 0loss/FBt3/assignment_loss‚0.4872112415703896 .loss/FRt2/detection_loss‚2.470674549755843 /loss/FRt4/assignment_loss‚4.133459482644056 1loss/SRqqt3/assignment_loss‚2.307544690267793 1loss/FBt2/assignment_loss‚0.44776309188455343 1loss/SRqqt3/detection_loss‚2.5048105264532157 .loss/FBt2/detection_loss‚3.179199233651161 0loss/FRt1/assignment_loss‚1.9853053712377362 /loss/FRt3/assignment_loss‚4.002987885475159 1loss/SRqqt4/detection_loss‚1.8867454522965001 .loss/FBt4/detection_loss‚2.532242293541248 % _timestamp‚1.7884302867777026e+09Õ k•o’ý‚(:‡ ×åÔû¯™ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚94.901' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°Úþ‚(:’ ×åÔîŒ÷ disk.sda2.out‚0 disk.rbd2.out‚ 79.1025390625 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0' proc.memory.availableMB‚ 950.2890625 disk.sda3.in‚0 disk.dm-1.out‚ 81.921875" disk.nvme0n1.out‚ 11309.0703125 disk.rbd4.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 network.recv‚752276 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2870.1953125* proc.memory.percent‚17.518281936645508 cpu‚25.345291910730182 disk.sda3.out‚ 81.921875 disk.nvme0c0n1.in‚0 proc.cpu.threads‚17 network.sent‚ 5.042374e+06 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚ 11309.0703125 disk.sda.out‚ 81.921875 disk.rbd1.out‚ 1.068359375) disk./.usagePercent‚73.83659040113473% disk./.usageGB‚1100.2185554504395 disk.drbd1777.out‚0 disk.rbd4.in‚0$ disk.nvme0c0n1.out‚ 11309.0703125$ memory_percent‚94.19989585876465êUçG“ÿ‚(:ˆ æåÔ“ºú gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.186' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p‚fö'€‚(:œ æåÔц§ disk.rbd2.in‚0 disk.sda2.out‚0 disk.drbd1777.out‚0( proc.memory.availableMB‚ 950.20703125 disk.rbd4.out‚0 disk.rbd1.out‚ 1.1005859375 disk.rbd0.out‚0 disk.rbd2.out‚79.19287109375 cpu‚25.343116589132737 network.recv‚757655 disk.rbd4.in‚0 network.sent‚ 5.1501e+06% disk./.usageGB‚1100.2186584472656 disk.sda3.out‚ 82.3203125" proc.memory.rssMB‚ 2870.1953125) disk./.usagePercent‚73.83659731333815 disk.dm-0.in‚0 disk.sda.out‚ 82.3203125 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚11310.087890625 disk.rbd1.in‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0! disk.dm-0.out‚11310.087890625 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.518281936645508 disk.sda1.out‚0 disk.dm-1.out‚ 82.3203125 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚11310.087890625 disk.sda2.in‚0 disk.rbd3.out‚0$ memory_percent‚94.20039653778076›B+·2- _wandbruntime‚2006  _runtime‚2006‚40ÓÇ‚Á _runtime‚2006.461231508 _step‚214 & lr-AdamW/pg1‚0.0002169697292826936 & lr-AdamW/pg2‚0.0002169697292826936 trainer/global_step‚2649 % _timestamp‚1.7884303217564628e+09QÎØÌƒÆ & lr-AdamW/pg1‚0.0002169697292826936 & lr-AdamW/pg2‚0.0002169697292826936 trainer/global_step‚2649 % _timestamp‚1.7884303217564628e+09 _runtime‚2006.461231508 _step‚214ÖÛ€h2„- _wandbruntime‚2006  _runtime‚2006bygÞÙ'…ÓO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50019135 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12750208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170496349263360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2329075896832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88510650 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38511554 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153993049061888 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50019135 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168167428602368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218036733952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218693095936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚2006.462253353 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149958660107264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2328920660992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2210524334893 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38511554 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156211067792896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152169184442157 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38511552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12749772 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156211067792896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153993031058944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22470406 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88530689 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149958643067904 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153993031058944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170496528439296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚215 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884303217572913e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38511552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50019135 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168167428602368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218036733952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218693095936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚2006.462253353 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149958660107264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2328920660992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2210524334893 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38511554 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156211067792896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152169184442157 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38511552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12749772 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156211067792896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153993031058944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22470406 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88530689 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149958643067904 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153993031058944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170496528439296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚215 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884303217572913e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38511552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152224905007889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50037615 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88563185 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38525568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152224233530569 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88543120 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2218856161792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88543120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156268914637824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154049384097792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2329940615168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22478574 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38525570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168229079340032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2649 _step‚216 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38525568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50017552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154049402100736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35232809 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150013547198464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154049402100736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219512537088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38525570 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35233244 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170559043895296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928ZÃöç&‰áM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156268914637824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154049384097792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2329940615168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22478574 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38525570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168229079340032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2649 _step‚216 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38525568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50017552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154049402100736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35232809 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150013547198464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154049402100736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219512537088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38525570 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35233244 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170559043895296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152224905007889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50037615 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88563185 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38525568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152224233530569 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88543120 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2218856161792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88543120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065Ø!S5N2Š- _wandbruntime‚2007  _runtime‚2007k,ÚW‹Ñ %loss/total_loss‚51.09266466729479 /loss/FRt2/detection_loss‚2.3012473939784934 2loss/SRqqt1/assignment_loss‚1.3187359716633664 2loss/SRqqt2/assignment_loss‚1.6391377099182294 2loss/SRqqt4/assignment_loss‚1.9146496368428836 /loss/FRt1/detection_loss‚2.1855289003915255 /loss/FBt1/detection_loss‚0.5697633940735831 epoch‚2 trainer/global_step‚2649 0loss/FBt3/assignment_loss‚0.4580180930003621 /loss/FRt3/detection_loss‚2.7732228545042186 .loss/FRt4/detection_loss‚2.086085667428763 0loss/SRqqt1/detection_loss‚1.148181402727085 /loss/FBt2/detection_loss‚3.8309850096702576 _step‚217 /loss/FRt1/assignment_loss‚2.508171856403351 0loss/SRqqt3/detection_loss‚2.861357021790284 /loss/FRt3/assignment_loss‚4.787961220129942 2loss/SRqqt3/assignment_loss‚2.2192655626254587 % _timestamp‚1.7884303224435186e+09 .loss/FRt2/assignment_loss‚3.45744969270059 0loss/FBt4/assignment_loss‚0.7173101486904281 1loss/SRqqt4/detection_loss‚1.9434309257910802 .loss/FBt3/detection_loss‚3.197850356499354 0loss/FRt4/assignment_loss‚3.7693247315676315 1loss/SRqqt2/detection_loss‚2.2206301015356313 _runtime‚2007.148446812 1loss/FBt1/assignment_loss‚0.21381549345568754 0loss/FBt2/assignment_loss‚0.5853760987517566 .loss/FBt4/detection_loss‚2.385165423154831ž‡Ÿ×\ŒÖ .loss/FRt2/assignment_loss‚3.45744969270059 0loss/FBt4/assignment_loss‚0.7173101486904281 1loss/SRqqt4/detection_loss‚1.9434309257910802 .loss/FBt3/detection_loss‚3.197850356499354 0loss/FRt4/assignment_loss‚3.7693247315676315 1loss/SRqqt2/detection_loss‚2.2206301015356313 _runtime‚2007.148446812 1loss/FBt1/assignment_loss‚0.21381549345568754 0loss/FBt2/assignment_loss‚0.5853760987517566 .loss/FBt4/detection_loss‚2.385165423154831 %loss/total_loss‚51.09266466729479 /loss/FRt2/detection_loss‚2.3012473939784934 2loss/SRqqt1/assignment_loss‚1.3187359716633664 2loss/SRqqt2/assignment_loss‚1.6391377099182294 2loss/SRqqt4/assignment_loss‚1.9146496368428836 /loss/FRt1/detection_loss‚2.1855289003915255 /loss/FBt1/detection_loss‚0.5697633940735831 epoch‚2 trainer/global_step‚2649 0loss/FBt3/assignment_loss‚0.4580180930003621 /loss/FRt3/detection_loss‚2.7732228545042186 .loss/FRt4/detection_loss‚2.086085667428763 0loss/SRqqt1/detection_loss‚1.148181402727085 /loss/FBt2/detection_loss‚3.8309850096702576 _step‚217 /loss/FRt1/assignment_loss‚2.508171856403351 0loss/SRqqt3/detection_loss‚2.861357021790284 /loss/FRt3/assignment_loss‚4.787961220129942 2loss/SRqqt3/assignment_loss‚2.2192655626254587 % _timestamp‚1.7884303224435186e+09Ù‰;¨ß“‚(:ˆ õåÔŽ£Ž gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.368' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.0736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A{Ù)Ž‚(:ž õåÔÿÊ®’ network.recv‚762241 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚0 disk.nvme0c0n1.in‚0* proc.memory.percent‚17.518281936645508 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚79.20849609375 disk.dm-0.in‚0% disk./.usageGB‚1099.1583557128906 disk.sda2.out‚0 disk.rbd4.in‚0 disk.sda.out‚ 82.9140625$ memory_percent‚94.20197010040283" proc.memory.rssMB‚ 2870.1953125 network.sent‚ 5.156307e+06 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd4.out‚0! disk.dm-0.out‚11409.197265625$ disk.nvme0n1.out‚11409.197265625( proc.memory.availableMB‚ 949.94921875 disk.rbd0.in‚0& disk.nvme0c0n1.out‚11409.197265625 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda.in‚0 disk.rbd1.in‚0) disk./.usagePercent‚73.76543950718099 disk.sda1.out‚0 disk.dm-1.out‚ 82.9140625 disk.sda3.out‚ 82.9140625 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 1.1005859375 cpu‚25.340903916614156*æv‚(:’ „åÔëÌÒ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚141.616' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.323200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãëÞó.‚(:£ „åÔÌ°î) disk./.usagePercent‚73.76538984171935 disk.rbd4.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚0 disk.sda.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.518281936645508 network.recv‚768180$ disk./.usageGB‚1099.157615661621 disk.sda2.out‚0" disk.dm-0.out‚11575.7216796875 disk.sda.out‚ 83.59765625 disk.rbd1.out‚ 1.16357421875 cpu‚25.33884067623358 network.sent‚ 5.264315e+06 disk.sda2.in‚0 disk.sda3.out‚ 83.59765625 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2870.1953125 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚11575.7216796875 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚79.27099609375 disk.dm-1.in‚0 disk.dm-1.out‚ 83.59765625% disk.nvme0n1.out‚11575.7216796875( proc.memory.availableMB‚ 950.14453125 disk.rbd0.out‚0 disk.nvme0n1.in‚0$ memory_percent‚94.20077800750732'ôž‘‚(:“ “åÔø‰ø gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.274800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íü¯3’‚(:¨ “åÔ—ÈÑ network.sent‚ 5.270892e+06 disk.sda.out‚ 84.12109375 disk.dm-1.out‚ 84.12109375% disk./.usageGB‚1099.1575965881348 disk.sda3.in‚0 disk.rbd4.in‚0$ memory_percent‚94.20077800750732 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2870.1953125 cpu‚25.336723795908547 network.recv‚772652) disk./.usagePercent‚73.76538856168168 disk.sda3.out‚ 84.12109375 disk.rbd3.out‚0# disk.dm-0.out‚11576.03759765625 disk.rbd1.out‚ 1.16357421875 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚11576.03759765625 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚11576.03759765625 disk.sda2.out‚0* proc.memory.percent‚17.518281936645508 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.dm-0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 950.14453125 disk.rbd2.out‚80.79833984375 disk.drbd1777.in‚0 proc.cpu.threads‚17~(@Õ2“- _wandbruntime‚2042  _runtime‚2042—`Z>ɔà ' lr-AdamW/pg1‚0.00021629828270942937 ' lr-AdamW/pg2‚0.00021629828270942937 trainer/global_step‚2699 % _timestamp‚1.7884303578891633e+09 _runtime‚2042.593918236 _step‚218ïú½Î•È ' lr-AdamW/pg1‚0.00021629828270942937 ' lr-AdamW/pg2‚0.00021629828270942937 trainer/global_step‚2699 % _timestamp‚1.7884303578891633e+09 _runtime‚2042.593918236 _step‚218Úâ±ë 2–- _wandbruntime‚2042  _runtime‚2042Þ½V—ÓO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156810701004288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39212352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159070379906048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22878804 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154954554832453 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2259678901760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156810683001344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12976057 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2371793629696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2251359674301 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2259022539776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35854439 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159070379906048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159069705541120 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39212354 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90157541 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39212354 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156810701004288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12975635 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154955226309565 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171249965485568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35854863 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22878806 % _timestamp‚1.7884303578899014e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173621944582656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159069705541120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2250705236549 trainer/global_step‚2699 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2371955156992 _rutP|Ø ntime‚2042.594843991 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2259678901760 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35854863 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22878806 % _timestamp‚1.7884303578899014e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173621944582656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159069705541120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2250705236549 trainer/global_step‚2699 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2371955156992 _runtime‚2042.594843991 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2259678901760 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35854439 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159070379906048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112Û×£2™- _wandbruntime‚2043  _runtime‚2043x½'ã&šÝM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90169972 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259841967616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22886972 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12980566 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260498342912 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚152758753726464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884303585493448e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚220 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173684257908736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159127552386048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171311640163328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚171311616223232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2260498342912 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50943604 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156867054043136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156867036040192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90190037 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173684451751424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90190037 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156867054043136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39226370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50963667 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50943604 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152758770765824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155010946875297 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39226368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35867094 _runtime‚2043.254782395 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50963667 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155010275397977 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90169972 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35867540 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259841967616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2372641685504 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39226370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2251521671513 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2699 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156867036040192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2372811588096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517š´a($›âM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260498342912 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚152758753726464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884303585493448e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚220 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173684257908736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159127552386048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171311640163328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚171311616223232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2260498342912 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50943604 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156867054043136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156867036040192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90190037 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173684451751424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90190037 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156867054043136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39226370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50963667 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50943604 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152758770765824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155010946875297 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39226368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35867094 _runtime‚2043.254782395 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50963667 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155010275397977 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90169972 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35867540 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259841967616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2372641685504 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39226370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2251521671513 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2699 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156867036040192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2372811588096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90169972 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.cÚ¨ÝCÖurrent‚ 674378240 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259841967616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22886972 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12980566 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844ÜK­|2œ- _wandbruntime‚2043  _runtime‚2043qR¯ETÎ .loss/FRt2/detection_loss‚2.587649766095849 0loss/FBt2/assignment_loss‚0.4119230918586254 /loss/FBt3/detection_loss‚3.4058103362719216 # _timestamp‚1.78843035855157e+09 /loss/FRt3/detection_loss‚3.3047878543535867 1loss/SRqqt1/detection_loss‚0.8323669286595808 .loss/FBt2/detection_loss‚3.682247906923294 trainer/global_step‚2699 _step‚221 2loss/SRqqt2/assignment_loss‚1.8220501497704933 /loss/FBt1/detection_loss‚0.5360101660571589 %loss/total_loss‚49.89517493694368 0loss/FRt1/assignment_loss‚1.6312983619670074 /loss/FRt3/assignment_loss‚4.669650773207347 2loss/SRqqt4/assignment_loss‚1.9018772134184838 /loss/FBt3/assignment_loss‚0.289324441821173 2loss/SRqqt1/assignment_loss‚1.0707913039604555 1loss/SRqqt3/assignment_loss‚2.434606185555458 1loss/SRqqt3/detection_loss‚2.6666378458340962 1loss/SRqqt4/detection_loss‚1.8744053602218629 _runtime‚2043.256219884 0loss/FBt4/assignment_loss‚0.6517080848033612 .loss/FRt1/detection_loss‚1.831363822799176 .loss/FRt2/assignment_loss‚3.67216605918352 1loss/FBt1/assignment_loss‚0.25588357346747775 .loss/FRt4/detection_loss‚2.442018318300446 1loss/SRqqt2/detection_loss‚2.0271018407446273 .loss/FBt4/detection_loss‚2.006906367265261 epoch‚2 0loss/FRt4/assignment_loss‚3.8865891844034195‘c†°YžÓ 2loss/SRqqt2/assignment_loss‚1.8220501497704933 /loss/FBt1/detection_loss‚0.5360101660571589 %loss/total_loss‚49.89517493694368 0loss/FRt1/assignment_loss‚1.6312983619670074 /loss/FRt3/assignment_loss‚4.669650773207347 2loss/SRqqt4/assignment_loss‚1.9018772134184838 /loss/FBt3/assignment_loss‚0.289324441821173 2loss/SRqqt1/assignment_loss‚1.0707913039604555 1loss/SRqqt3/assignment_loss‚2.434606185555458 1loss/SRqqt3/detection_loss‚2.6666378458340962 1loss/SRqqt4/detection_loss‚1.8744053602218629 _runtime‚2043.256219884 0loss/FBt4/assignment_loss‚0.6517080848033612 .loss/FRt1/detection_loss‚1.831363822799176 .loss/FRt2/assignment_loss‚3.67216605918352 1loss/FBt1/assignment_loss‚0.25588357346747775 .loss/FRt4/detection_loss‚2.442018318300446 1loss/SRqqt2/detection_loss‚2.0271018407446273 .loss/FBt4/detection_loss‚2.006906367265261 epoch‚2 0loss/FRt4/assignment_loss‚3.8865891844034195 .loss/FRt2/detection_loss‚2.587649766095849 0loss/FBt2/assignment_loss‚0.4119230918586254 /loss/FBt3/detection_loss‚3.4058103362719216 # _timestamp‚1.78843035855157e+09 /loss/FRt3/detection_loss‚3.3047878543535867 1loss/SRqqt1/detection_loss‚0.8323669286595808 .loss/FBt2/detection_loss‚3.682247906923294 trainer/global_step‚2699 _step‚221ÝC Uä“Ÿ‚(:ˆ ¢åÔ¢ÎûŽ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.292' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.4584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Ý( ‚(: ¢åÔ Õ disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 84.58203125" proc.memory.rssMB‚ 2870.1953125$ disk./.usageGB‚1099.157642364502" disk.dm-0.out‚11576.4560546875 disk.sda.out‚ 84.58203125 disk.drbd1777.in‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.518281936645508 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚80.86083984375 disk.dm-1.in‚0% disk.nvme0n1.out‚11576.4560546875 disk.rbd1.out‚ 1.2265625' disk.nvme0c0n1.out‚11576.4560546875( disk./.usagePercent‚73.7653916337721 disk.rbd4.in‚0 disk.rbd0.out‚0 cpu‚25.334656084191543 disk.dm-0.in‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 950.328125 network.recv‚776401 disk.sda.in‚0$ memory_percent‚94.19965744018555 disk.sda3.in‚0 network.sent‚ 5.276071e+06 disk.dm-1.out‚ 84.58203125 disk.nvme0n1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.sda1.in‚0±høðž¡‚(:“ ±åÔ“ôƒŽ gpu.0.gpu‚33.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.264' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.652800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ÍjL¢‚(: ±åÔÀõ¢ disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625 disk.rbd2.out‚80.87646484375 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 network.recv‚783141 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0) disk./.usagePercent‚73.76539649791523 disk.rbd0.in‚0 disk.sda3.out‚ 85.15234375 disk.nvme0n1.out‚ 11577.03125* proc.memory.percent‚17.518281936645508 disk.rbd4.out‚0 disk.sda2.out‚0 disk.dm-0.out‚ 11577.03125 disk.sda1.in‚0 disk.dm-1.out‚ 85.15234375 disk.sda2.in‚0( proc.memory.availableMB‚ 949.55078125" proc.memory.rssMB‚ 2870.1953125 proc.cpu.threads‚17# disk./.usageGB‚1099.15771484375 disk.rbd3.in‚0 disk.sda.out‚ 85.15234375 cpu‚25.33262892836688 network.sent‚ 5.385049e+06 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚0" disk.nvme0c0n1.out‚ 11577.03125$ memory_percent‚94.20440196990967 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0¿"¬2£- _wandbruntime‚2077  _runtime‚2077Ä`ǤÁ trainer/global_step‚2749 % _timestamp‚1.7884303927965875e+09 _runtime‚ 2077.5013452 _step‚222 ' lr-AdamW/pg1‚0.00021560537382486866 ' lr-AdamW/pg2‚0.00021560537382486866{ÕÌ¥Æ ' lr-AdamW/pg1‚0.00021560537382486866 ' lr-AdamW/pg2‚0.00021560537382486866 trainer/global_step‚2749 % _timestamp‚1.7884303927965875e+09 _runtime‚ 2077.5013452 _step‚222Þ”&‰s2¦- _wandbruntime‚2077  _runtime‚2077 ö0Ù'§ÓO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159628334943744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39913154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36489435 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2300008345600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23287206 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884303927973552e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157741268176973 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51871239 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2300008345600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39913152 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176747369007616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155449073163264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51851202 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174332502368768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚2077.502267167 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36489860 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2414666491392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2292195013709 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161929017654272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2300664707584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2291540575957 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39913152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176747168860160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2414842698752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23287204 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13202654 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161929017654272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161928343289344 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174332526308864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13202231 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91764354 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159628334943744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚2749 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36489860 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2414666491392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2292195013709 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161929017654272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2300664707584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2291540575957 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39913152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176747168860160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2414842698752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23287204 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13202654 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161929017654272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161928343289344 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174332526308864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13202231 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91764354 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159628334943744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚2749 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161986190134272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39927168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51869656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36502577 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161986190134272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39927170 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176809866886656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91796824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91816889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161985515756032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174394153106432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2078.150979459 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91796824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23295372 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157796317265385 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51889719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13207203 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.largeMŒð[_pool.allocated‚155503977293824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159684687982592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174394177046528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36502136 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884303934455473e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39927170 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159684687982592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176809677238272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2293011448881 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2415689840128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2301484148736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39927168 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51869656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2300827773440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51889719 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159684705985536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚159684705985536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2301484148736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23295374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91816889 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884303934455473e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39927170 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159684687982592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176809677238272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2293011448881 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2415689840128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2301484148736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39927168 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51869656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2300827773440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51889719 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159684705985536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚159684705985536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2301484148736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23295374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91816889 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161986190134272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39927168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51869656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36502577 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161986190134272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39927170 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176809866886656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91796824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91816889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161985515756032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174394153106432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2078.150979459 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91796824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23295372 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157796317265385 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51889719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13207203 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155503977293824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159684687982592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174394177046528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36502136 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164786981037568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160527310044381 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158194279691264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93411245 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2340994151424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162445986886144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2332375915365 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164787655402496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162446004889088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40613954 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915904 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52797291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164786981037568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13429693 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 trainer/global_step‚2799 _step‚227 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2341650513408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13430136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179872625271296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52797291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23695606 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40613954 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884304280593367e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52777254 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160526638567269 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 65443775221)¸ØO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2112.764257535 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2341650513408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23695604 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40613952 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162445986886144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164786981037568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160527310044381 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158194279691264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93411245 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2340994151424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162445986886144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2332375915365 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164787655402496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162446004889088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40613954 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915904 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52797291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164786981037568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13429693 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 trainer/global_step‚2799 _step‚227 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2341650513408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13430136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179872625271296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52797291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23695606 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40613954 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884304280593367e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52777254 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160526638567269 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2333846788289 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2342469954560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37137907 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2333192350329 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162502357927936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2341813579264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52795708 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179935134786048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93443741 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164844827882496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93423676 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162502339924992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40627970 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162502357927936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚228 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40627968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179935322337280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40627968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23703774 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13434586 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884304287136369e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177476689989632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40627970 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2341813579264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2342469954560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160583030610113 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93443741 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164844827882496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93423676 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162502339924992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40627970 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162502357927936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚228 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40627968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179935322337280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40627968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23703774 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13434586 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884304287136369e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177476689989632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40627970 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2341813579264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2342469954560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160583030610113 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2333846788289 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2342469954560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37137907 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2333192350329 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162502357927936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2341813579264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52795708 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179935134786048ägLp2¼- _wandbruntime‚2113  _runtime‚2113î[½Õ 0loss/FRt1/assignment_loss‚1.3182586867894446 1loss/SRqqt4/detection_loss‚1.9444641014751123 /loss/FBt1/detection_loss‚0.6725805086582393 2loss/SRqqt2/assignment_loss‚1.6279019427796204 _step‚229 /loss/FBt4/detection_loss‚2.1817062041338753 epoch‚2 % _timestamp‚1.7884304287151985e+09 0loss/FBt4/assignment_loss‚0.5771181428914561 %loss/total_loss‚47.91435594963876 2loss/SRqqt1/assignment_loss‚1.0381123946137625 1loss/SRqqt3/detection_loss‚2.7019308218732476 /loss/FBt2/detection_loss‚3.2197876572608948 trainer/global_step‚2799 /loss/FRt3/assignment_loss‚4.237685738187848 0loss/FBt1/assignment_loss‚0.2661437515712312 .loss/FRt2/detection_loss‚2.337992299510085 /loss/FRt3/detection_loss‚2.8597501877582436 .loss/FRt4/detection_loss‚2.081785354208439 1loss/SRqqt1/detection_loss‚0.9397119860330673 /loss/FRt2/assignment_loss‚3.522802039333012 /loss/FRt4/assignment_loss‚4.901146906487485 2loss/SRqqt3/assignment_loss‚2.2181999469175935 2loss/SRqqt4/assignment_loss‚1.1515912050191237 .loss/FBt3/detection_loss‚3.643613249063492 1loss/SRqqt2/detection_loss‚1.9726412256558736 1loss/FBt2/assignment_loss‚0.49341630066434544 1loss/FBt3/assignment_loss‚0.23258072230787263 /loss/FRt1/detection_loss‚1.7734345764453923 _runtime‚2113.419865597Vÿ+Ä`¾Ú 1loss/FBt3/assignment_loss‚0.23258072230787263 /loss/FRt1/detection_loss‚1.7734345764453923 _runtime‚2113.419865597 0loss/FRt1/assignment_loss‚1.3182586867894446 1loss/SRqqt4/detection_loss‚1.9444641014751123 /loss/FBt1/detection_loss‚0.6725805086582393 2loss/SRqqt2/assignment_loss‚1.6279019427796204 _step‚229 /loss/FBt4/detection_loss‚2.1817062041338753 epoch‚2 % _timestamp‚1.7884304287151985e+09 0loss/FBt4/assignment_loss‚0.5771181428914561 %loss/total_loss‚47.91435594963876 2loss/SRqqt1/assignment_loss‚1.0381123946137625 1loss/SRqqt3/detection_loss‚2.7019308218732476 /loss/FBt2/detection_loss‚3.2197876572608948 trainer/global_step‚2799 /loss/FRt3/assignment_loss‚4.237685738187848 0loss/FBt1/assignment_loss‚0.2661437515712312 .loss/FRt2/detection_loss‚2.337992299510085 /loss/FRt3/detection_loss‚2.8597501877582436 .loss/FRt4/detection_loss‚2.081785354208439 1loss/SRqqt1/detection_loss‚0.9397119860330673 /loss/FRt2/assignment_loss‚3.522802039333012 /loss/FRt4/assignment_loss‚4.901146906487485 2loss/SRqqt3/assignment_loss‚2.2181999469175935 2loss/SRqqt4/assignment_loss‚1.1515912050191237 .loss/FBt3/detection_loss‚3.643613249063492 1loss/SRqqt2/detection_loss‚1.9726412256558736 1loss/FBt2/assignment_loss‚0.49341630066434544å½Z… “¿‚(:ˆ ÞåÔ³æÿ gpu.0.gpu‚43.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qÌ×À‚(: ÞåÔ¯ø‘ disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.75392326426072 disk.rbd2.out‚ 82.685546875 disk.sda.out‚ 86.54296875$ memory_percent‚97.09696769714355% disk./.usageGB‚1098.9867553710938 disk.rbd1.in‚0 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚0" disk.dm-0.out‚11580.0654296875 network.recv‚796450 disk.dm-1.in‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚11580.0654296875 disk.sda2.in‚0 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda.in‚0* proc.memory.percent‚17.518281936645508 cpu‚25.32654841240755 disk.sda3.in‚0 disk.rbd4.out‚0" proc.memory.rssMB‚ 2870.1953125 disk.rbd1.out‚ 1.2265625' disk.nvme0c0n1.out‚11580.0654296875W=Ðà™ disk.sda3.out‚ 86.54296875 network.sent‚ 5.504378e+06 disk.dm-1.out‚ 86.54296875 disk.rbd2.in‚0' proc.memory.availableMB‚ 475.6328125’bôƒ“Á‚(:ˆ íåÔ©î…Ž gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.087' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6174 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†Ëc!‚(:– íåÔŽŸˆ‘ disk.nvme0n1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.75392940844154 disk.rbd3.out‚0% disk./.usageGB‚1098.9868469238281 disk.rbd2.out‚ 82.685546875 disk.rbd4.out‚0 disk.dm-0.out‚11580.63671875# disk.nvme0n1.out‚11580.63671875 disk.rbd1.in‚0 network.recv‚803322 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 87.1796875 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 87.1796875$ memory_percent‚97.09951877593994* proc.memory.percent‚17.518281936645508 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 87.1796875 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.rbd1.out‚ 1.2265625" proc.memory.rssMB‚ 2870.1953125 network.sent‚ 5.613389e+06 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 475.21484375 cpu‚25.324628327965478% disk.nvme0c0n1.out‚11580.63671875 disk.dm-0.in‚0ÿü۞Â(:“ üåÔ±í gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.797' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.559399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠÝüÄ‚(:Œ üåÔ—ÁŸ‘ disk.sda2.out‚0 disk.rbd2.in‚0% disk./.usageGB‚1099.1168899536133 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 11714.2265625 disk.rbd3.in‚0 disk.sda.out‚ 87.640625 disk.rbd1.out‚ 1.2265625 disk.nvme0c0n1.in‚0 cpu‚25.322726633596755 disk.sda3.out‚ 87.640625 disk.drbd1777.out‚0 disk.rbd0.out‚0( disk./.usagePercent‚73.7626567052838 disk.dm-0.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 82.763671875 network.recv‚807286$ memory_percent‚97.09806442260742 disk.drbd1777.in‚0 disk.dm-0.out‚ 11714.2265625 disk.dm-1.in‚0" disk.nvme0n1.out‚ 11714.2265625& proc.memory.availableMB‚ 475.453125" proc.memory.rssMB‚ 2870.1953125 proc.cpu.threads‚17 network.sent‚ 5.61883e+06* proc.memory.percent‚17.518281936645508 disk.sda3.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 87.640625 disk.sda2.in‚0 &‹v2Å- _wandbruntime‚2148  _runtime‚2148+M½`ÉÆà ' lr-AdamW/pg1‚0.00021415575116917019 ' lr-AdamW/pg2‚0.00021415575116917019 trainer/global_step‚2849 % _timestamp‚1.7884304635605438e+09 _runtime‚2148.265297183 _step‚230/‚¬RÎÇÈ _step‚230 ' lr-AdamW/pg1‚0.00021415575116917019 ' lr-AdamW/pg2‚0.00021415575116917019 trainer/global_step‚2849 % _timestamp‚1.7884304635605438e+09 _runtime‚2148.265297183æ‰ÒÇ2È- _wandbruntime‚2148  _runtime‚2148K˜‚<Ø'ÉÒO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2382636319232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24104006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180497600075264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165263656831488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚167646293150720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53703306 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2382636319232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2500757365248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160939469179904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2381979957248 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41314754 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41314752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13658466 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95018058 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167646293150720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37762034 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163312680434677 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2373211254773 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165263656831488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165263638828544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2373865692525 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884304635612633e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180497576135168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167645618785792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2381979957248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41314754 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182998510579200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163313351911789 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24104004 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚231 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13658030 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53703306 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚167645618785792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53723343 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2500910503936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41314754 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41314752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13658466 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95018058 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167646293150720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37762034 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163312680434677 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2373211254773 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165263656831488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165263638828544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2373865692525 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884304635612633e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180497576135168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167645618785792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2381979957248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41314754 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182998510579200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163313351911789 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24104004 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚231 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13658030 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53703306 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚167645618785792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53723343 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2500910503936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0çý¥^B2Ë- _wandbruntime‚2148  _runtime‚2148 Ï%äÁ!ÌÝM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183061027635712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚2148.941188533 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37774684 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160994390349824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚2849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53741823 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13662957 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884304642358677e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95050528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41328770 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53721760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2382799385088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41328770 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2374027689737 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚165320009870336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2374682127697 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37775131 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2501606920192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95050528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95070593 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41328768 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24112172 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180559250812928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167702791252480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165319991867392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚165320009870336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95070593 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163368401000201 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53721760 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41328768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180559226872832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163369072477521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚09/óàè&ÍâM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚2148.941188533 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37774684 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160994390349824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚2849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53741823 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13662957 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884304642358677e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95050528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41328770 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53721760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2382799385088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41328770 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2374027689737 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚165320009870336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2374682127697 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37775131 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2501606920192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95050528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95070593 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41328768 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24112172 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180559250812928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167702791252480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165319991867392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚165320009870336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95070593 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163368401000201 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53721760 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41328768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180559226872832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163369072477521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183061027635712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928èa<õ2Î- _wandbruntime‚2148  _runtime‚21488s÷IWÏÑ .loss/FRt2/detection_loss‚2.897884367090283 .loss/FBt2/detection_loss‚4.784234444300334 /loss/FBt4/detection_loss‚2.8498694234424167 trainer/global_step‚2849 1loss/SRqqt4/assignment_loss‚1.761609306037426 .loss/FRt3/detection_loss‚2.841238019795253 0loss/FRt2/assignment_loss‚3.3241860396934277 /loss/FRt3/assignment_loss‚4.225075939605976 1loss/FBt3/assignment_loss‚0.41011373615927166 0loss/FBt4/assignment_loss‚0.6753911094533068 /loss/FRt1/detection_loss‚1.9265844775570764 /loss/FRt4/detection_loss‚2.2848000607142844 _runtime‚ 2148.94266231 0loss/FRt1/assignment_loss‚1.6135496536890666 1loss/SRqqt2/assignment_loss‚1.684428497372816 /loss/FBt1/detection_loss‚0.5624086641047983 /loss/FBt3/detection_loss‚3.2285920646455555 1loss/FBt1/assignment_loss‚0.24828045558715905 epoch‚2 % _timestamp‚1.7884304642376678e+09 1loss/SRqqt3/assignment_loss‚1.869337792459287 0loss/SRqqt1/detection_loss‚1.059942263827631 0loss/SRqqt2/detection_loss‚2.037450914581617 0loss/FRt4/assignment_loss‚3.6360582672059536 0loss/FBt2/assignment_loss‚0.2307644858956337 1loss/SRqqt3/detection_loss‚2.3402620883364427 1loss/SRqqt4/detection_loss‚1.7841763913631439 %loss/total_loss‚49.25376063588327 2loss/SRqqt1/assignment_loss‚0.9775221729651093 _step‚2332ó¿\ÐÖ 1loss/SRqqt4/assignment_loss‚1.761609306037426 .loss/FRt3/detection_loss‚2.841238019795253 0loss/FRt2/assignment_loss‚3.3241860396934277 /loss/FRt3/assignment_loss‚4.225075939605976 1loss/FBt3/assignment_loss‚0.41011373615927166 0loss/FBt4/assignment_loss‚0.6753911094533068 /loss/FRt1/detection_loss‚1.9265844775570764 /loss/FRt4/detection_loss‚2.2848000607142844 _runtime‚ 2148.94266231 0loss/FRt1/assignment_loss‚1.6135496536890666 1loss/SRqqt2/assignment_loss‚1.684428497372816 /loss/FBt1/detection_loss‚0.5624086641047983 /loss/FBt3/detection_loss‚3.2285920646455555 1loss/FBt1/assignment_loss‚0.24828045558715905 epoch‚2 % _timestamp‚1.7884304642376678e+09 1loss/SRqqt3/assignment_loss‚1.869337792459287 0loss/SRqqt1/detection_loss‚1.059942263827631 0loss/SRqqt2/detection_loss‚2.037450914581617 0loss/FRt4/assignment_loss‚3.6360582672059536 0loss/FBt2/assignment_loss‚0.2307644858956337 1loss/SRqqt3/detection_loss‚2.3402620883364427 1loss/SRqqt4/detection_loss‚1.7841763913631439 %loss/total_loss‚49.25376063588327 2loss/SRqqt1/assignment_loss‚0.9775221729651093 _step‚233 .loss/FRt2/detection_loss‚2.897884367090283 .loss/FBt2/detection_loss‚4.784234444300334 /loss/FBt4/detection_loss‚2.8498694234424167 trainer/global_step‚2849韬û0žÑ‚(:“ ‹‘åÔë¥Ý gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.186599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â¿'Ò‚(:œ ‹‘åÔ¼‚©‘ network.sent‚ 5.625046e+06 network.recv‚812048% disk./.usageGB‚1099.2469291687012 disk.drbd1777.in‚0 disk.nvme0n1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0( proc.memory.availableMB‚ 475.17578125 disk.sda3.out‚ 88.171875 disk.dm-1.out‚ 88.171875& disk.nvme0n1.out‚11848.17822265625( disk.nvme0c0n1.out‚11848.17822265625 disk.rbd1.out‚ 1.2265625$ memory_percent‚97.09975719451904" proc.memory.rssMB‚ 2870.1953125) disk./.usagePercent‚73.77138374611852 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.518281936645508 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 88.171875 disk.nvme0c0n1.in‚0 disk.sda2.in‚0 disk.rbd2.out‚ 82.763671875 disk.rbd3.in‚0 disk.dm-0.in‚0 disk.sda.in‚0 cpu‚25.320739074225962 proc.cpu.threads‚17# disk.dm-0.out‚11848.17822265625÷—žÓ‚(:“ š‘åÔÄ×ä gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚141.452' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.290399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ݼÍ,Ô‚(:¡ š‘åÔð¯ö network.sent‚ 5.731863e+06& disk.nvme0n1.out‚11848.43212890625 disk.nvme0c0n1.in‚0% disk./.usageGB‚1099.2461051940918 disk.nvme0n1.in‚0 disk.rbd2.in‚0 disk.rbd4.out‚0 cpu‚25.318903278120082 disk.sda3.in‚0 disk.sda.out‚ 88.5390625) disk./.usagePercent‚73.77132844849112 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625 disk.sda3.out‚ 88.5390625 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.sda.in‚0* proc.memory.percent‚17.518281936645508 network.recv‚817276 disk.dm-1.out‚ 88.5390625 disk.sda2.in‚0 disk.rbd2.out‚82.86962890625 disk.drbd1777.out‚0# disk.dm-0.out‚11848.43212890625 disk.sda2.out‚0 disk.rbd4.in‚0$ memory_percent‚97.09765911102295( proc.memory.availableMB‚ 475.51953125" proc.memory.rssMB‚ 2870.1953125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚11848.43212890625 disk.rbd0.out‚0§Âj½2Õ- _wandbruntime‚2185  _runtime‚2185ßû)ÉÖà ' lr-AdamW/pg1‚0.00021339933949722107 ' lr-AdamW/pg2‚0.00021339933949722107 trainer/global_step‚2899 % _timestamp‚1.7884305010072212e+09 _runtime‚2185.711995859 _step‚234\=FÎ×È % _timestamp‚1.7884305010072212e+09 _runtime‚2185.711995859 _step‚234 ' lr-AdamW/pg1‚0.00021339933949722107 ' lr-AdamW/pg2‚0.00021339933949722107 trainer/global_step‚2899ê%6&Ã2Ø- _wandbruntime‚2185  _runtime‚2185S¨DÙ'ÙÓO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38398989 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96664949 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166099393779197 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚235 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2543990997504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2423622125056 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2414046594181 % _timestamp‚1.7884305010079112e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54649395 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54629358 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168081308773888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174110208 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170504930898944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24512406 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170504256534016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚183580136958464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163684675707904 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54629358 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54649395 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170504930898944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186123929905664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166098722302085 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96644910 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170504256534016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2899 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2543816887296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42015554 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2414701031933 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 _runtime‚2185.712820694 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186124127955968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2422965763072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24512404 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13886135 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42015552 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168081308773888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38398539 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198050304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168081290770944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170504256534016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚183580136958464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163684675707904 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54629358 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54649395 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170504930898944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186123929905664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166098722302085 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96644910 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170504256534016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2899 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceu³’ñ™StatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2543816887296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42015554 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2414701031933 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 _runtime‚2185.712820694 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186124127955968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2422965763072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24512404 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13886135 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42015552 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168081308773888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38398539 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198050304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168081290770944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2543990997504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2423622125056 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2414046594181 % _timestamp‚1.7884305010079112e+09ë!]¿2Û- _wandbruntime‚2186  _runtime‚2186t}2Çã&ÜÝM trainer/global_step‚2899 _runtime‚2186.378861484 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96677380 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38411779 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24520572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186186636567552 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2423785190912 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54667875 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2423785190912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚183641763756032 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42029568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163739579838464 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186186451113472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54647812 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96697445 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚183641787696128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168137661812736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163739596877824 % _timestamp‚1.7884305016736772e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38411332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168137643809792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168137661812736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2424441566208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96677380 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42029568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42029570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513984 _step‚236 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170561429000704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170562103378944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944þåhè&ÝâM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96677380 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38411779 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24520572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186186636567552 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2423785190912 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54667875 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2423785190912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚183641763756032 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42029568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163739579838464 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186186451113472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54647812 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96697445 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚183641787696128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168137661812736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163739596877824 % _timestamp‚1.7884305016736772e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38411332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168137643809792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168137661812736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2424441566208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96677380 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42029568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42029570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513984 _step‚236 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170561429000704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170562103378944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚2899 _runtime‚2186.378861484 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414ì½Ä´`2Þ- _wandbruntime‚2186  _runtime‚2186@±¦OWßÑ 1loss/SRqqt2/detection_loss‚1.7381999593514663 -loss/FBt2/detection_loss‚4.26674211025238 /loss/FBt3/detection_loss‚2.7108676433563232 trainer/global_step‚2899 _step‚237 /loss/FRt3/assignment_loss‚4.355786947103647 1loss/SRqqt2/assignment_loss‚2.043130476658161 2loss/SRqqt3/assignment_loss‚1.8242755913072162 /loss/FRt1/detection_loss‚1.7158710953959249 % _timestamp‚1.7884305016752303e+09 0loss/SRqqt1/assignment_loss‚1.29732378938219 0loss/FBt2/assignment_loss‚0.5951058864593506 0loss/FBt4/assignment_loss‚0.5245439271227672 1loss/SRqqt1/detection_loss‚1.2261501716678602 1loss/SRqqt4/detection_loss‚1.9044424059256069 /loss/FBt1/detection_loss‚1.0374609552353369 /loss/FBt4/detection_loss‚1.9107862619253306 /loss/FRt1/assignment_loss‚2.206364531266062 0loss/FRt2/assignment_loss‚3.6923162566041046 /loss/FRt4/assignment_loss‚4.036293195353614 1loss/FBt1/assignment_loss‚0.18674388904628178 .loss/FRt4/detection_loss‚2.121575609842936 _runtime‚2186.379891846 2loss/SRqqt4/assignment_loss‚1.4887079025073997 0loss/FBt3/assignment_loss‚0.5078308794647427 .loss/FRt2/detection_loss‚2.052452319073227 1loss/SRqqt3/detection_loss‚2.4650222924020557 &loss/total_loss‚48.975729369808434 epoch‚2 /loss/FRt3/detection_loss‚3.0677352731044474k8ù¼\àÖ epoch‚2 /loss/FRt3/detection_loss‚3.0677352731044474 1loss/SRqqt2/detection_loss‚1.7381999593514663 -loss/FBt2/detection_loss‚4.26674211025238 /loss/FBt3/detection_loss‚2.7108676433563232 trainer/global_step‚2899 _step‚237 /loss/FRt3/assignment_loss‚4.355786947103647 1loss/SRqqt2/assignment_loss‚2.043130476658161 2loss/SRqqt3/assignment_loss‚1.8242755913072162 /loss/FRt1/detection_loss‚1.7158710953959249 % _timestamp‚1.7884305016752303e+09 0loss/SRqqt1/assignment_loss‚1.29732378938219 0loss/FBt2/assignment_loss‚0.5951058864593506 0loss/FBt4/assignment_loss‚0.5245439271227672 1loss/SRqqt1/detection_loss‚1.2261501716678602 1loss/SRqqt4/detection_loss‚1.9044424059256069 /loss/FBt1/detection_loss‚1.0374609552353369 /loss/FBt4/detection_loss‚1.9107862619253306 /loss/FRt1/assignment_loss‚2.206364531266062 0loss/FRt2/assignment_loss‚3.6923162566041046 /loss/FRt4/assignment_loss‚4.036293195353614 1loss/FBt1/assignment_loss‚0.18674388904628178 .loss/FRt4/detection_loss‚2.121575609842936 _runtime‚2186.379891846 2loss/SRqqt4/assignment_loss‚1.4887079025073997 0loss/FBt3/assignment_loss‚0.5078308794647427 .loss/FRt2/detection_loss‚2.052452319073227 1loss/SRqqt3/detection_loss‚2.4650222924020557 &loss/total_loss‚48.975729369808434í:狼’á‚(:‡ ©‘åÔâæè gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.579' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.5158 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~Å^Äâ‚(:‹ ©‘åÔј¯ disk.nvme0n1.out‚ 11848.875 disk.sda2.out‚0 network.recv‚821843) disk./.usagePercent‚73.77133894480002 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.out‚ 11848.875 disk.sda3.out‚ 89.08203125 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.dm-0.in‚0 disk.sda.in‚0 disk.sda.out‚ 89.08203125 disk.sda2.in‚0 disk.rbd2.out‚82.86962890625 cpu‚25.317096131025007 disk.sda1.out‚0 disk.rbd1.out‚ 1.2265625 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0$ memory_percent‚97.09610939025879" proc.memory.rssMB‚ 2870.1953125 network.sent‚ 5.738155e+06 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 89.08203125 disk.nvme0n1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1099.2462615966797 disk.drbd1777.in‚0 disk.dm-0.out‚ 11848.875 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 475.7734375* proc.memory.percent‚17.518281936645508Y*úŠ“ã‚(:ˆ ¸‘åÔÌùë gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹(ÈDä‚(:‰ ¸‘åÔ„ûº‘ network.sent‚ 5.846128e+06 disk.rbd0.out‚0 disk.rbd2.out‚84.52197265625 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 89.82421875 disk.nvme0n1.out‚ 11857.625 disk.dm-0.in‚0 disk.sda1.in‚0 network.recv‚827745 disk.nvme0c0n1.in‚0 disk.nvme0c0n1.out‚ 11857.625 disk.drbd1777.in‚0 disk.rbd3.in‚0) disk./.usagePercent‚73.77189268909659 disk.rbd2.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625% proc.memory.availableMB‚ 475.96875" proc.memory.rssMB‚ 2870.1953125* proc.memory.percent‚17.518281936645508 disk.rbd4.out‚0 disk.dm-0.out‚ 11857.625 disk.sda3.out‚ 89.82421875% disk./.usageGB‚1099.2545127868652 disk.sda3.in‚0$ memory_percent‚97.09491729736328 cpu‚25.315274808560286 disk.sda.out‚ 89.82421875 disk.sda2.out‚0 disk.rbd3.out‚0ÌÃåU’å‚(:‡ Ç‘åÔ‹ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.555' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.511 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)n6(æ‚(: Ç‘åÔïã¤( disk.nvme0c0n1.out‚11861.35400390625 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚11861.35400390625 disk.sda.out‚ 90.4296875 disk.dm-1.out‚ 90.4296875 disk.sda2.in‚0 disk.rbd2.out‚84.52197265625 disk.rbd1.in‚0" proc.memory.rssMB‚ 2870.1953125* proc.memory.percent‚17.518281936645508 network.recv‚831980% disk./.usageGB‚1099.2573547363281 disk.dm-0.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0 cpu‚25.313419420074087 disk.sda3.in‚0 disk.sda3.out‚ 90.4296875% proc.memory.availableMB‚ 475.96875 network.sent‚ 5.852332e+06( disk./.usagePercent‚73.7720834147096 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.rbd1.out‚ 1.2265625 disk.sda2.out‚0& disk.nvme0n1.out‚11861.35400390625$ memory_percent‚97.09491729736328 disk.rbd4.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0Þ$}C2ç- _wandbruntime‚2221  _runtime‚2221DbÉYÈè $ _timestamp‚1.788430536447629e+09 _runtime‚2221.152362246 _step‚238 ' lr-AdamW/pg1‚0.00021262206964939583 ' lr-AdamW/pg2‚0.00021262206964939583 trainer/global_step‚2949Ã@èµéÇ ' lr-AdamWÓ;ÒŸ»/pg1‚0.00021262206964939583 ' lr-AdamW/pg2‚0.00021262206964939583 trainer/global_step‚2949 $ _timestamp‚1.788430536447629e+09 _runtime‚2221.152362246 _step‚238î\Ð1=2ê- _wandbruntime‚2221  _runtime‚2221ˆï'Ù'ëÓO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42716354 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2463951568896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55575447 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195953152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166429882235904 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173362894282240 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55575447 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170898942713344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189249646102528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168884764169493 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚239 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98271762 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173363568647168 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚173362894282240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 % _timestamp‚1.7884305364484956e+09 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170898960716288 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2464607930880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2464607930880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55555410 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39035596 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24920806 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24920804 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55555410 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170898942713344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2464607930880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2464607930880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55555410 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39035596 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24920806 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24920804 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55555410 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170898942713344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173362894282240 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55575447 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170898942713344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189249646102528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168884764169493 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚239 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98271762 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173363568647168 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚173362894282240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 % _timestamp‚1.7884305364484956e+09ï§['2í- _wandbruntime‚2221  _runtime‚2221ç³Ç·ã&îÝM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39048348 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999744 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170955313755136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168940484735017 % _timestamp‚1.7884305371068244e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39048820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14119846 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168941156212337 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170955295752192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2465427372032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55593927 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2456352806513 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166484786366464 _step‚240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98304232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170955313755136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42730368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98324297 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173420066748928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170955295752192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42730370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2465427372032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2587822788608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚2221.812326354 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42730368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14119376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186724300639232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24928974 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2464770996736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173420741127168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189312147367936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55593927 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98304232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42730370 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2587650788864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166484803405824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24928972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55573864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939840 trainer/global_step‚2949 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98324297 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚470 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98324297 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173420066748928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170955295752192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42730370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2465427372032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2587822788608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚2221.812326354 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42730368 REDeviceStatsMonitor.on_train_batcõOƒ°Òh_end/inactive_split.small_pool.freed‚14119376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186724300639232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24928974 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2464770996736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173420741127168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189312147367936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55593927 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98304232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42730370 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2587650788864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166484803405824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24928972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55573864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939840 trainer/global_step‚2949 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98324297 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚470 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39048348 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999744 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170955313755136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168940484735017 % _timestamp‚1.7884305371068244e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39048820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14119846 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168941156212337 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170955295752192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2465427372032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55593927 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2456352806513 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166484786366464 _step‚240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98304232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170955313755136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42730368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ðX9¨Á2ð- _wandbruntime‚2221  _runtime‚2221–H[ñÕ 0loss/FRt4/assignment_loss‚3.8067116141319275 1loss/SRqqt2/detection_loss‚1.7690323169808835 2loss/SRqqt2/assignment_loss‚1.5521390817593783 epoch‚2 0loss/FRt1/assignment_loss‚2.1805108723541102 1loss/SRqqt1/assignment_loss‚1.243264243284713 /loss/FRt3/detection_loss‚2.7215085593429773 1loss/SRqqt3/detection_loss‚2.5675531458109617 % _timestamp‚1.7884305371091745e+09 _runtime‚2221.813825319 _step‚241 1loss/FBt4/assignment_loss‚0.36692689005285456 .loss/FRt2/detection_loss‚2.231104712486267 2loss/SRqqt4/assignment_loss‚1.6406506916807433 /loss/FBt3/detection_loss‚3.2006598114967346 %loss/total_loss‚48.79611644940533 0loss/FBt1/assignment_loss‚0.2313123959312738 1loss/FBt2/assignment_loss‚0.24613039443890253 /loss/FRt1/detection_loss‚2.1349180246082446 1loss/SRqqt1/detection_loss‚1.1994455758093008 .loss/FBt2/detection_loss‚4.038691163063049 trainer/global_step‚2949 /loss/FRt3/assignment_loss‚4.418647785444517 .loss/FRt4/detection_loss‚2.356915806978941 1loss/SRqqt4/detection_loss‚2.0064671241811345 /loss/FBt1/detection_loss‚0.5832865071128981 2loss/SRqqt3/assignment_loss‚2.0464292597025633 0loss/FBt3/assignment_loss‚0.7372378971838063 /loss/FBt4/detection_loss‚2.1119507193565368 /loss/FRt2/assignment_loss‚3.404621856212616KwÊ_`òÚ /loss/FRt1/detection_loss‚2.1349180246082446 1loss/SRqqt1/detection_loss‚1.1994455758093008 .loss/FBt2/detection_loss‚4.038691163063049 trainer/global_step‚2949 /loss/FRt3/assignment_loss‚4.418647785444517 .loss/FRt4/detection_loss‚2.356915806978941 1loss/SRqqt4/detection_loss‚2.0064671241811345 /loss/FBt1/detection_loss‚0.5832865071128981 2loss/SRqqt3/assignment_loss‚2.0464292597025633 0loss/FBt3/assignment_loss‚0.7372378971838063 /loss/FBt4/detection_loss‚2.1119507193565368 /loss/FRt2/assignment_loss‚3.404621856212616 0loss/FRt4/assignment_loss‚3.8067116141319275 1loss/SRqqt2/detection_loss‚1.7690323169808835 2loss/SRqqt2/assignment_loss‚1.5521390817593783 epoch‚2 0loss/FRt1/assignment_loss‚2.1805108723541102 1loss/SRqqt1/assignment_loss‚1.243264243284713 /loss/FRt3/detection_loss‚2.7215085593429773 1loss/SRqqt3/detection_loss‚2.5675531458109617 % _timestamp‚1.7884305371091745e+09 _runtime‚2221.813825319 _step‚241 1loss/FBt4/assignment_loss‚0.36692689005285456 .loss/FRt2/detection_loss‚2.231104712486267 2loss/SRqqt4/assignment_loss‚1.6406506916807433 /loss/FBt3/detection_loss‚3.2006598114967346 %loss/total_loss‚48.79611644940533 0loss/FBt1/assignment_loss‚0.2313123959312738 1loss/FBt2/assignment_loss‚0.24613039443890253ñ%7óžó‚(:“ Ö‘åÔ›ìüŽ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.266' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.453199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôv ³,ô‚(:¡ Ö‘åÔˆŠß disk.sda3.in‚0 disk.drbd1777.in‚0# disk.dm-0.out‚11861.67041015625& disk.nvme0n1.out‚11861.67041015625 disk.sda2.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0( disk.nvme0c0n1.out‚11861.67041015625 disk.rbd0.out‚0 network.sent‚ 5.857613e+06 disk.dm-0.in‚0 disk.sda2.out‚0 cpu‚25.31166109097016 disk.rbd2.in‚0 disk.rbd2.out‚ 84.6865234375 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd4.out‚0 disk.rbd1.out‚ 1.2265625 disk.nvme0c0n1.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 476.1484375% disk./.usageGB‚1099.2574348449707 disk.sda.in‚0 disk.sda.out‚ 90.79296875 disk.dm-1.in‚0 network.recv‚835702) disk./.usagePercent‚73.77208879086781 disk.sda3.out‚ 90.79296875 disk.rbd4.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 90.79296875$ memory_percent‚97.09382057189941* proc.memory.percent‚17.518281936645508" proc.memory.rssMB‚ 2870.1953125 proc.cpu.threads‚17ø%Éï“õ‚(:ˆ å‘åÔºžÐ gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚107.709' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y.÷)ö‚(:ž å‘åÔŸêß disk.rbd4.in‚0 disk.sda.out‚ 91.3984375 disk.rbd0.in‚0( proc.memory.availableMB‚ 476.01171875 network.recv‚842088) disk./.usagePercent‚73.78046842947856 disk.sda1.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.518305778503418 disk.sda3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 1.2265625' disk.nvme0c0n1.out‚11961.1279296875 cpu‚25.309849669651857 network.sent‚ 5.966353e+06% disk./.usageGB‚1099.3822975158691 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 91.3984375 disk.sda2.in‚0 disk.sda3.out‚ 91.3984375% disk.nvme0n1.out‚11961.1279296875 disk.rbd2.in‚0$ memory_percent‚97.09465503692627 proc.cpu.threads‚17" disk.dm-0.out‚11961.1279296875 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2870.19921875 disk.rbd4.out‚0 disk.nvme0n1.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 84.6865234375 ÛD‚2÷- _wandbruntime‚2256  _runtime‚2256øÃO`Éøà ' lr-AdamW/pg1‚0.00021182410360756575 ' lr-AdamW/pg2‚0.00021182410360756575 trainer/global_step‚2999 % _timestamp‚1.7884305716222513e+09 _runtime‚2256.327037408 _step‚242Œò§áÎùÈ _step‚242 ' lr-AdamW/pg1‚0.00021182410360756575 ' lr-AdamW/pg2‚0.00021182410360756575 trainer/global_step‚2999 % _timestamp‚1.7884305716222513e+09 _runtime‚2256.327037408òŽ/ü2ú- _wandbruntime‚2256  _runtime‚2256ÜkNºØ'ûÒO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176222206395392 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99918653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14343082 _runtime‚ 2256.32810519 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2505593736704 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56501499 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629701062656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚192374887847424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2504937374720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169175105803264 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192375075411968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43417154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39672715 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189745210724864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171670806036901 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189745186784768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2505593736704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99898614 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56481462 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43417152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176221532030464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2629864687104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚243`H”Z† ü×O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99898614 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56481462 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43417152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176221532030464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2629864687104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚243 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176222206395392 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99918653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14343082 _runtime‚ 2256.32810519 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2505593736704 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56501499 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629701062656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚192374887847424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2504937374720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169175105803264 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192375075411968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43417154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39672715 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189745210724864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171670806036901 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189745186784768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2505593736704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414ó¥åô2ý- _wandbruntime‚2257  _runtime‚2257òx=lã&þÝM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169229992894464 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 _step‚244 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14347747 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56519979 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99931084 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2630728366080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171726526602425 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2505756802560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2505756802560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173772965697536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189806861462528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39685119 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171727198079745 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169230009933824 % _timestamp‚1.7884305723058734e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99931084 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2630562657792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56519979 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14348180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43431170 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43431168 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56499916 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚192437589828608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2497188145921 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173772947694592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2506413177856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2506413177856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99951149 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173772965697536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176279378875392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192437400180224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚2257.011303836 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43431168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25337372 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0WvC¿è&ÿâM PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2506413177856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2506413177856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99951149 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173772965697536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176279378875392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192437400180224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚2257.011303836 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43431168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25337372 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169229992894464 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 _step‚244 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14347747 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56519979 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99931084 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2630728366080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171726526602425 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2505756802560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2505756802560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173772965697536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189806861462528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39685119 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171727198079745 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169230009933824 % _timestamp‚1.7884305723058734e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99931084 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2630562657792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56519979 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14348180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43431170 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43431168 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56499916 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚192437589828608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2497188145921 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173772947694592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944ô²=A±2€- _wandbruntime‚2257  _runtime‚2257¬Ó°à^Ø .loss/FBt4/detection_loss‚2.576677998671165 %loss/total_loss‚48.48569955714592 /loss/FRt3/assignment_loss‚4.010914596644315 2loss/SRqqt2/assignment_loss‚1.8462965877032746 1loss/FBt2/assignment_loss‚0.44312697369605303 1loss/FBt3/assignment_loss‚0.34734334067131084 2loss/SRqqt1/assignment_loss‚1.2372751384973526 1loss/SRqqt3/detection_loss‚2.3946369263258847 /loss/FBt1/detection_loss‚0.7294571201280267 0loss/FBt1/assignment_loss‚0.2463947732869662 % _timestamp‚1.7884305723076494e+09 0loss/FRt1/assignment_loss‚1.9483213636610244 /loss/FRt4/assignment_loss‚3.325581362434462 0loss/FBt4/assignment_loss‚0.3525809900691876 /loss/FRt1/detection_loss‚1.9418849674363932 /loss/FBt2/detection_loss‚3.4768183827400208 _runtime‚2257.012719032 _step‚245 0loss/FRt2/assignment_loss‚3.2903410358862444 /loss/FRt3/detection_loss‚3.3445115468718787 /loss/FRt4/detection_loss‚2.3740725593239653 1loss/SRqqt4/detection_loss‚2.1824650156497953 2loss/SRqqt3/assignment_loss‚2.1266283497891645 /loss/FRt2/detection_loss‚2.1630634741349652 1loss/SRqqt2/detection_loss‚1.9533452765614379 epoch‚2 trainer/global_step‚2999 2loss/SRqqt4/assignment_loss‚2.1331867069005965 1loss/SRqqt1/detection_loss‚1.0352731093477745 /loss/FBt3/detection_loss‚3.0055019607146582m´~§c‚Ý /loss/FBt1/detection_loss‚0.7294571201280267 0loss/FBt1/assignment_loss‚0.2463947732869662 % _timestamp‚1.7884305723076494e+09 0loss/FRt1/assignment_loss‚1.9483213636610244 /loss/FRt4/assignment_loss‚3.325581362434462 0loss/FBt4/assignment_loss‚0.3525809900691876 /loss/FRt1/detection_loss‚1.9418849674363932 /loss/FBt2/detection_loss‚3.4768183827400208 _runtime‚2257.012719032 _step‚245 0loss/FRt2/assignment_loss‚3.2903410358862444 /loss/FRt3/detection_loss‚3.3445115468718787 /loss/FRt4/detection_loss‚2.3740725593239653 1loss/SRqqt4/detection_loss‚2.1824650156497953 2loss/SRqqt3/assignment_loss‚2.1266283497891645 /loss/FRt2/detection_loss‚2.1630634741349652 1loss/SRqqt2/detection_loss‚1.9533452765614379 epoch‚2 trainer/global_step‚2999 2loss/SRqqt4/assignment_loss‚2.1331867069005965 1loss/SRqqt1/detection_loss‚1.0352731093477745 /loss/FBt3/detection_loss‚3.0055019607146582 .loss/FBt4/detection_loss‚2.576677998671165 %loss/total_loss‚48.48569955714592 /loss/FRt3/assignment_loss‚4.010914596644315 2loss/SRqqt2/assignment_loss‚1.8462965877032746 1loss/FBt2/assignment_loss‚0.44312697369605303 1loss/FBt3/assignment_loss‚0.34734334067131084 2loss/SRqqt1/assignment_loss‚1.2372751384973526 1loss/SRqqt3/detection_loss‚2.3946369263258847õ9W#“ƒ‚(:ˆ ô‘åÔçÂî gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚161.186' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Uñ)„‚(:ž ô‘åÔ¼õ" disk.dm-0.out‚12060.2490234375 disk.dm-1.in‚0% disk.nvme0n1.out‚12060.2490234375* proc.memory.percent‚17.518305778503418 cpu‚25.30824914889507 disk.sda.out‚ 91.83984375 disk.sda1.out‚0 disk.dm-1.out‚ 91.83984375 disk.nvme0n1.in‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 1.2265625' disk.nvme0c0n1.out‚12060.2490234375 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2870.19921875 network.recv‚846110) disk./.usagePercent‚73.78886112447357 disk.drbd1777.in‚0 disk.nvme0c0n1.in‚0 disk.sda3.out‚ 91.83984375& proc.memory.availableMB‚ 476.109375 network.sent‚ 5.971842e+06% disk./.usageGB‚1099.5073547363281 disk.rbd2.out‚ 84.8271484375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.sda3.in‚0$ memory_percent‚97.09405899047852 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0UÄäe“…‚(:ˆ ƒ’åÔùŠ™Ž gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`Ó †‚(:’ ƒ’åÔž›Œ‘$ memory_percent‚97.09558486938477$ disk./.usageGB‚1099.758373260498 disk.sda.out‚92.4375 disk.dm-1.in‚0 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 475.859375 cpu‚25.306655494325963 network.recv‚852540% disk.nvme0n1.out‚12060.8876953125 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.rbd4.out‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0) disk./.usagePercent‚73.80570718824919 disk.dm-1.out‚92.4375 disk.sda2.out‚0 disk.rbd2.out‚ 86.3388671875 disk.sda3.in‚0* proc.memory.percent‚17.518305778503418" disk.dm-0.out‚12060.8876953125 disk.nvme0c0n1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2870.19921875 network.sent‚ 6.080616e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 1.2265625 disk.sda3.out‚92.4375 disk.dm-0.in‚0 disk.sda.in‚0' disk.nvme0c0n1.out‚12060.8876953125 disk.rbd0.out‚0 disk.drbd1777.out‚0šuk2‡- _wandbruntime‚2292  _runtime‚2292AaáXȈ _runtime‚2292.153160276 _step‚246 ' lr-AdamW/pg1‚0.00021100560766665794 ' lr-AdamW/pg2‚0.00021100560766665794 trainer/global_step‚3049 $ _timestamp‚1.788430607448298e+096TÞÂÍ‰Ç ' lr-AdamW/pg2‚0.00021100560766665794 trainer/global_step‚3049 $ _timestamp‚1.788430607448298e+09 _runtime‚2292.153160276 _step‚246 ' lr-AdamW/pg1‚0.00021100560766665794öšn92Š- _wandbruntime‚2292  _runtime‚2292!~ä‹×O N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101545505 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171920295291904 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 % _timestamp‚1.7884306074493988e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179080844143616 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2672773139968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57407514 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40309104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2546579542528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192827747608064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57427551 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179080844143616 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176534264601088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚3049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14571950 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174456847904309 _step‚247 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174457519381421 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171920312331264 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101525466 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44117954 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14571500 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179080169778688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batchk FHù_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2546579542528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176534264601088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44117954 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176534246598144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44117952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195500684372480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2537207050157 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57407514 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25737604 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179080169778688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57427551 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195500496807936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25737606 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44117952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2545923180544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2672936764416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2536552612405 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192827723667968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179080844143616 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176534264601088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚3049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14571950 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174456847904309 _step‚247 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174457519381421 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171920312331264 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101525466 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44117954 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14571500 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179080169778688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2546579542528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176534264601088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44117954 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176534246598144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44117952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195500684372480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2537207050157 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57407514 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25737604 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179080169778688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57427551 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195500496807936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25737606 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44117952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2545923180544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2672936764416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2536552612405 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192827723667968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101578001 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44131968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101557936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14576018 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44131970 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192889374405632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 $ _timestamp‚1.788430608150887e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179138016623616 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101557936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25745772 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44131968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2547398983680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57446031 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176590599636992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚3049 _runtime‚2292.856301215 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2546742608384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171975216461824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2673791803904 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2673621901312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171975199422464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192889398345728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174512568469833 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44131970 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2546742608384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179138016623616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57425968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57425968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176590599636992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2547398983680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195562996306944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2538023485329 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40321790 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40322241 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176590617639936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101578001 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2537369047369 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179137342245376 3õ(êåM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57425968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176590599636992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2547398983680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195562996306944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2538023485329 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40321790 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40322241 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176590617639936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101578001 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2537369047369 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179137342245376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101578001 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44131968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101557936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14576018 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44131970 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192889374405632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 $ _timestamp‚1.788430608150887e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179138016623616 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101557936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25745772 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44131968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2547398983680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57446031 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_trd»éö ain_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176590599636992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚3049 _runtime‚2292.856301215 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2546742608384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171975216461824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2673791803904 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2673621901312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171975199422464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192889398345728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174512568469833 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44131970 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2546742608384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179138016623616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57425968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0øž‡ é2- _wandbruntime‚2292  _runtime‚2292Ñw+W‘Ñ _step‚249 1loss/SRqqt3/detection_loss‚2.3141813641414046 _runtime‚2292.857408411 2loss/SRqqt3/assignment_loss‚2.6772868055850267 0loss/FBt1/assignment_loss‚0.2542668860506408 .loss/FRt3/detection_loss‚3.527830219268799 .loss/FRt4/detection_loss‚2.645474715743746 0loss/SRqqt2/detection_loss‚1.869273250395397 0loss/FBt4/assignment_loss‚0.5148434309909717 /loss/FRt2/detection_loss‚2.3653432519539543 /loss/FBt1/detection_loss‚1.1195949949559598 -loss/FBt2/detection_loss‚5.47201281785965 -loss/FBt3/detection_loss‚4.02571976184845 /loss/FBt4/detection_loss‚2.9595991720755896 2loss/SRqqt2/assignment_loss‚1.6690638596719165 1loss/FBt2/assignment_loss‚0.04990199208054946 0loss/FBt3/assignment_loss‚0.3515028342604637 1loss/SRqqt1/detection_loss‚1.0495382085668319 % _timestamp‚1.7884306081527653e+09 /loss/FRt1/detection_loss‚1.7640536011507113 trainer/global_step‚3049 /loss/FRt2/assignment_loss‚3.137538422708926 /loss/FRt3/assignment_loss‚4.020463323593139 0loss/FRt4/assignment_loss‚3.6767248000417436 2loss/SRqqt1/assignment_loss‚1.1441420676243124 2loss/SRqqt4/assignment_loss‚1.3985815239497101 1loss/SRqqt4/detection_loss‚2.0378668033159695 epoch‚2 0loss/FRt1/assignment_loss‚1.5216338286797206 %loss/total_loss‚51.56643793651358 h<\’Ö trainer/global_step‚3049 /loss/FRt2/assignment_loss‚3.137538422708926 /loss/FRt3/assignment_loss‚4.020463323593139 0loss/FRt4/assignment_loss‚3.6767248000417436 2loss/SRqqt1/assignment_loss‚1.1441420676243124 2loss/SRqqt4/assignment_loss‚1.3985815239497101 1loss/SRqqt4/detection_loss‚2.0378668033159695 epoch‚2 0loss/FRt1/assignment_loss‚1.5216338286797206 %loss/total_loss‚51.56643793651358 _step‚249 1loss/SRqqt3/detection_loss‚2.3141813641414046 _runtime‚2292.857408411 2loss/SRqqt3/assignment_loss‚2.6772868055850267 0loss/FBt1/assignment_loss‚0.2542668860506408 .loss/FRt3/detection_loss‚3.527830219268799 .loss/FRt4/detection_loss‚2.645474715743746 0loss/SRqqt2/detection_loss‚1.869273250395397 0loss/FBt4/assignment_loss‚0.5148434309909717 /loss/FRt2/detection_loss‚2.3653432519539543 /loss/FBt1/detection_loss‚1.1195949949559598 -loss/FBt2/detection_loss‚5.47201281785965 -loss/FBt3/detection_loss‚4.02571976184845 /loss/FBt4/detection_loss‚2.9595991720755896 2loss/SRqqt2/assignment_loss‚1.6690638596719165 1loss/FBt2/assignment_loss‚0.04990199208054946 0loss/FBt3/assignment_loss‚0.3515028342604637 1loss/SRqqt1/detection_loss‚1.0495382085668319 % _timestamp‚1.7884306081527653e+09 /loss/FRt1/detection_loss‚1.7640536011507113ùzE©’“‚(:‡ ’’åÔÕ«× gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚101.075' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.215 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â Ëó$”‚(:™ ’’åÔ¥ù disk.sda2.in‚0 disk.nvme0c0n1.in‚0 network.sent‚ 6.085857e+06 disk.dm-1.out‚ 92.921875 cpu‚25.304892321585907 disk.rbd4.out‚0" disk.dm-0.out‚12159.9033203125 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 92.921875 disk.sda2.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.518305778503418 proc.cpu.threads‚17' disk.nvme0c0n1.out‚12159.9033203125 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚0% disk.nvme0n1.out‚12159.9033203125 disk.rbd1.out‚ 1.2265625# memory_percent‚97.0954418182373 network.recv‚855897 disk.sda3.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2870.19921875% disk./.usageGB‚1099.7574310302734 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 86.4169921875 disk.nvme0n1.in‚0' proc.memory.availableMB‚ 475.8828125) disk./.usagePercent‚73.80564395438822 disk.rbd2.in‚0 disk.sda3.out‚ 92.921875 disk.rbd4.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0³ðÂq“•‚(:ˆ ¡’åÔ‘ÚÞ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.909' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.1818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢~ì­/–‚(:¤ ¡’åÔ©±ù disk.rbd4.in‚0 disk.rbd4.out‚0 disk.dm-0.in‚0( disk.nvme0c0n1.out‚12259.34912109375 disk.rbd2.out‚ 86.4169921875 disk.drbd1777.in‚0 disk.rbd3.out‚0 network.recv‚862842) disk./.usagePercent‚73.80563883423754% disk./.usageGB‚1099.7573547363281 disk.dm-1.out‚ 93.72265625 disk.drbd1777.out‚0 network.sent‚ 6.194822e+06 disk.sda.out‚ 93.72265625 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.518305778503418# disk.dm-0.out‚12259.34912109375# proc.memory.rssMB‚ 2870.19921875 cpu‚25.303170903679813 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 93.72265625 disk.rbd3.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚12259.34912109375 disk.rbd1.out‚ 1.2265625( proc.memory.availableMB‚ 475.71484375 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚0 disk.rbd0.in‚0$ memory_percent‚97.09646701812744 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0?³%¯‘—‚(:† °’åÔ·ö  gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.97' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ}^!˜‚(:– °’åÔú¢¸ cpu‚25.301544538598012 network.recv‚866997( disk./.usagePercent‚73.8056401142752 disk.sda.out‚ 94.296875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚86.52685546875& proc.memory.availableMB‚ 475.953125% disk./.usageGB‚1099.7573738098145 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 94.296875 disk.rbd3.in‚0 disk.rbd4.in‚0* proc.memory.percent‚17.518305778503418 disk.nvme0n1.in‚0& disk.nvme0c0n1.out‚12358.466796875 disk.sda3.in‚0 disk.drbd1777.out‚0! disk.dm-0.out‚12358.466796875 disk.sda1.in‚0 disk.rbd1.out‚ 1.2265625 disk.nvme0c0n1.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd4.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2870.19921875$ disk.nvme0n1.out‚12358.466796875$ memory_percent‚97.09501266479492 network.sent‚ 6.200287e+06 disk.sda1.out‚0 disk.dm-1.out‚ 94.296875ª½ 72™- _wandbruntime‚2327  _runtime‚2327àCÕɚà ' lr-AdamW/pg1‚0.00021016675239999964 ' lr-AdamW/pg2‚0.00021016675239999964 trainer/global_step‚3099 % _timestamp‚1.7884306428150952e+09 _runtime‚2327.519835786 _step‚250E¶æÿÎ›È % _timestamp‚1.7884306428150952e+09 _runtime‚2327.519835786 _step‚250 ' lr-AdamW/pg1‚0.00021016675239999964 ' lr-AdamW/pg2‚0.00021016675239999964 trainer/global_step‚3099ú6$¦è2œ- _wandbruntime‚2327  _runtime‚2327„ˆÝ'×O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44818754 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198626413449728 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195910284491264 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177242889771717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44818752 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58333566 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2586908986368 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58333566 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14798700 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2577387951813 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚179351916543488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2715969528320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181939481891840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174665501819904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44818752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44818754 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58353603 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26146006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181938807526912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40944704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2716128958464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103172357 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021üâ'žÜO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103172357 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44818754 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198626413449728 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195910284491264 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177242889771717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44818752 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58333566 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2586908986368 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58333566 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14798700 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2577387951813 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚179351916543488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2715969528320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181939481891840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174665501819904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44818752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44818754 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58353603 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26146006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181938807526912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40944704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2716128958464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430ûò>N2Ÿ- _wandbruntime‚2328  _runtime‚2328í,=\† áM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14803277 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44832770 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2587728414208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036992 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884306434700437e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44832770 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚252 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2588384789504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40957449 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚2328.175317472 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44832768 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179408269582336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2716985193984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103184788 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26154174 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2588384789504 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103204853 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44832768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174720422989824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181996654371840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177298610337241 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195971935228928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58372083 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181996654371840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198688722385920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2716811097088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174720405950464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103204853 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2587728414208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103184788 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14803732 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195971911288832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177299281814561 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179408251579392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚409579062ó`ì&¡æM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚2328.175317472 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44832768 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179408269582336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2716985193984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103184788 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26154174 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2588384789504 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103204853 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44832768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174720422989824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181996654371840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177298610337241 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195971935228928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58372083 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181996654371840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198688722385920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2716811097088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174720405950464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103204853 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2587728414208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103184788 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14803732 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195971911288832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177299281814561 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179408251579392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40957906 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14803277 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44832770 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2587728414208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036992 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884306434700437e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44832770 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚252 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2588384789504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40957449 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182169568485888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184797445275136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚184798119640064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182169550482944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182169550482944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201752076762624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚255 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59279655 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180028931639125 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2759093860864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177410708347904 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45519552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198992821374464 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26554406 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177410725387264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QIDeviceStaP¡×(ätsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2618877728973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41581680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26554404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15027274 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180029603116237 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984<”|zá'®ÛO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59259618 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2628551154176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26554406 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177410725387264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2618877728973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41581680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26554404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15027274 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180029603116237 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 $ _timestamp‚1.788430678171964e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59259618 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41581240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15026836 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104779170 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2627894792192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2759255388160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198992797434368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚184798119640064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚184797445275136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2627894792192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104799209 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59279655 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201751891295232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182169568485888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2628551154176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45519554 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182169568485888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184797445275136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚184798119640064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182169550482944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182169550482944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201752076762624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚255 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59279655 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180028931639125 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2759093860864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177410708347904 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45519552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198992821374464 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2628714220032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚184855292120064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180084652204649 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41593875 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104831705 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2759953981440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180085323681969 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2619694164145 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2629370595328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚256 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182225903521792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201814589704704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182225903521792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104831705 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104811640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚184855292120064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199054472112128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚177465612478464 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59298135 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41594327 _runtime‚ 2363.53028346 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚184854617741824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2628714220032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199054448172032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182225921524736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59278072 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884306788250625e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45533570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2629370595328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2619039726185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚3149 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26562574 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15031303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201814402153472 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚3149 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26562574 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15031303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201814402153472 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2628714220032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚184855292120064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180084652204649 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41593875 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104831705 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2759953981440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180085323681969 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2619694164145 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2629370595328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚256 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182225903521792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201814589704704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182225903521792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104831705 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104811640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚184855292120064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199054472112128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚177465612478464 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59298135 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41594327 _runtime‚ 2363.53028346 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚184854617741824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2628714220032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199054448172032€„÷h 2²- _wandbruntime‚2363  _runtime‚2363¹Ü(2Õ³Ø /loss/FRt4/detection_loss‚2.1551656520113034 _step‚257 2loss/SRqqt4/assignment_loss‚2.0599918141961098 .loss/FRt2/detection_loss‚2.061104334280139 1loss/SRqqt3/detection_loss‚2.79711694}plí‰34744736 epoch‚2 _runtime‚2363.531425722 1loss/SRqqt1/detection_loss‚1.2291669581617628 % _timestamp‚1.7884306788267856e+09 0loss/FBt1/assignment_loss‚0.2223656062087675 2loss/FBt2/assignment_loss‚0.030361355049535632 /loss/FRt1/detection_loss‚1.5939835413642551 /loss/FBt2/detection_loss‚3.2160976827144623 trainer/global_step‚3149 /loss/FRt2/assignment_loss‚3.006278133783184 0loss/FRt3/assignment_loss‚3.8071012573857463 2loss/SRqqt1/assignment_loss‚1.0918310718876976 1loss/SRqqt4/detection_loss‚1.8288058885988199 /loss/FBt4/detection_loss‚2.3393755555152893 0loss/FRt1/assignment_loss‚1.7108643262282661 0loss/FRt4/assignment_loss‚3.4022406466463777 2loss/SRqqt2/assignment_loss‚1.7165555097865608 0loss/FBt4/assignment_loss‚0.5855678800832141 /loss/FBt1/detection_loss‚0.7017485826397133 &loss/total_loss‚46.359460250048315 1loss/SRqqt3/assignment_loss‚2.098327186087082 0loss/SRqqt2/detection_loss‚1.762776664405499 .loss/FBt3/detection_loss‚3.973248451948166 1loss/FBt3/assignment_loss‚0.04681956857287517 /loss/FRt3/detection_loss‚2.9225656390190125$ üêc´Ý 1loss/FBt3/assignment_loss‚0.04681956857287517 /loss/FRt3/detection_loss‚2.9225656390190125 /loss/FRt4/detection_loss‚2.1551656520113034 _step‚257 2loss/SRqqt4/assignment_loss‚2.0599918141961098 .loss/FRt2/detection_loss‚2.061104334280139 1loss/SRqqt3/detection_loss‚2.7971169434744736 epoch‚2 _runtime‚2363.531425722 1loss/SRqqt1/detection_loss‚1.2291669581617628 % _timestamp‚1.7884306788267856e+09 0loss/FBt1/assignment_loss‚0.2223656062087675 2loss/FBt2/assignment_loss‚0.030361355049535632 /loss/FRt1/detection_loss‚1.5939835413642551 /loss/FBt2/detection_loss‚3.2160976827144623 trainer/global_step‚3149 /loss/FRt2/assignment_loss‚3.006278133783184 0loss/FRt3/assignment_loss‚3.8071012573857463 2loss/SRqqt1/assignment_loss‚1.0918310718876976 1loss/SRqqt4/detection_loss‚1.8288058885988199 /loss/FBt4/detection_loss‚2.3393755555152893 0loss/FRt1/assignment_loss‚1.7108643262282661 0loss/FRt4/assignment_loss‚3.4022406466463777 2loss/SRqqt2/assignment_loss‚1.7165555097865608 0loss/FBt4/assignment_loss‚0.5855678800832141 /loss/FBt1/detection_loss‚0.7017485826397133 &loss/total_loss‚46.359460250048315 1loss/SRqqt3/assignment_loss‚2.098327186087082 0loss/SRqqt2/detection_loss‚1.762776664405499 .loss/FBt3/detection_loss‚3.973248451948166Ô\’µ‚(:‡ Ý’åÔ£•€Ž gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚88.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹@PÛ,¶‚(:¡ Ý’åÔèÜŠ‘ proc.cpu.threads‚17$ disk./.usageGB‚1100.257667541504 disk.dm-1.out‚ 96.22265625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0' proc.memory.availableMB‚ 475.6171875 cpu‚25.296910351973487) disk./.usagePercent‚73.83921524638329 disk.nvme0n1.in‚0 disk.rbd1.out‚ 1.2265625# disk.dm-0.out‚12557.85693359375# proc.memory.rssMB‚ 2870.19921875* proc.memory.percent‚17.518305778503418( disk.nvme0c0n1.out‚12557.85693359375 disk.sda2.out‚0 disk.sda3.out‚ 96.22265625 disk.rbd3.in‚0& disk.nvme0n1.out‚12557.85693359375 disk.dm-0.in‚0# memory_percent‚97.0970630645752 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd2.out‚88.12451171875 disk.sda3.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.sda.out‚ 96.22265625 network.recv‚881628 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0 network.sent‚ 6.32072e+06 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd4.out‚0³œIœ·‚(:‘ ì’åÔÛ¶Ú gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.54' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.907999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýÿ ((¸‚(: ì’åÔ¿¥š$ memory_percent‚97.09720611572266 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd4.in‚0" disk.dm-0.out‚12657.0068359375 network.sent‚ 6.428738e+06 network.recv‚887576 disk.sda3.out‚ 96.62890625 disk.rbd3.out‚0( disk./.usagePercent‚73.8392190864963 disk.sda2.out‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚12657.0068359375 disk.sda2.in‚0 disk.dm-0.in‚0* proc.memory.percent‚17.518305778503418 disk.nvme0c0n1.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 96.62890625 proc.cpu.threads‚17$ disk./.usageGB‚1100.257724761963 disk.rbd2.out‚88.28076171875 disk.rbd4.out‚0 disk.sda.in‚0 disk.sda.out‚ 96.62890625 disk.dm-1.in‚0# proc.memory.rssMB‚ 2870.19921875 disk.nvme0n1.in‚0' disk.nvme0c0n1.out‚12657.0068359375 disk.drbd1777.out‚0% proc.memory.availableMB‚ 475.59375 cpu‚25.295289709672378 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625 disk.sda3.in‚0OQ`2¹- _wandbruntime‚2397  _runtime‚2397À=©sɺà ' lr-AdamW/pg1‚0.00020842866736057428 ' lr-AdamW/pg2‚0.00020842866736057428 trainer/global_step‚3199 % _timestamp‚1.7884307132692537e+09 _runtime‚2397.973986529 _step‚258 Õ†Î»È trainer/global_step‚3199 % _timestamp‚1.7884307132692537e+09 _runtime‚2397.973986529 _step‚258 ' lr-AdamW/pg1‚0.00020842866736057428 ' lr-AdamW/pg2‚0.00020842866736057428‚ÓÈ°¿2¼- _wandbruntime‚2397  _runtime‚2397Kp¨OÝ'½×O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204877526140928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46220354 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2669536960000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184987202425344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187656757388288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2659713068381 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46220354 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46220352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60205707 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2669536960000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187656757388288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184987220428288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884307132699637e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚2397.974871817 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202075334317568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46220352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60205707 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15254366 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187656083023360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2802191823360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304512 _step‚259 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106406022 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180155931915264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2668880598016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26962806 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184987220428288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184987202425344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182815644983645 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60185670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15254826 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2668880598016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 J=â'¾ÜO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46220352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60205707 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15254366 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187656083023360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2802191823360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304512 _step‚259 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106406022 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180155931915264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2668880598016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26962806 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184987220428288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184987202425344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182815644983645 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60185670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15254826 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2668880598016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204877526140928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46220354 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2669536960000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184987202425344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187656757388288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2659713068381 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46220354 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46220352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60205707 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2669536960000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187656757388288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184987220428288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884307132699637e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚2397.974871817 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202075334317568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112ƒÒ 2¿- _wandbruntime‚2398  _runtime‚2398Ì|°ÛDÀáM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26970972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182870694072057 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2660529503553 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚185043573467136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚185043555464192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106458557 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26970974 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884307139322898e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18504357344—@£67136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202137008995328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2803048449024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187713255490048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204940033504256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180210819006464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42229839 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187713929868288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15258867 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60204124 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46234370 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106458557 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2659875065593 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60224187 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0M!„gì&ÁæM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46234368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60204124 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180210836045824 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚260 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2670356401152 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2660529503553 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚185043573467136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚185043555464192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106458557 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26970974 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884307139322898e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚185043573467136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202137008995328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2803048449024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187713255490048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204940033504256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180210819006464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42229839 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187713929868288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15258867 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60204124 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46234370 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106458557 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2659875065593 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60224187 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26970972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182870694072057 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205157895140864 _step‚263 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚182901138443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190514720771584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2845338584064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61131759 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42852411 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2845495917056 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61111722 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2709866403840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27371204 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187804854367744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205157871200768 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚190515395136512 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚2433.624516742 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187804854367744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15481654 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46921154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61111722 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚190514720771584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27371206 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2710522765824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2710522765824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42852860 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187804872370688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46921152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2709866403840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208003391057920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚3249 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61131759 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15481207 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108052913Ÿ™2âüÐÜO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15481207 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108052913 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚185601686851053 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46921152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46921154 UADeviceStatsMonitor.on_train_ba”ãŸæ#tch_start/requested_bytes.all.freed‚185601015373941 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205157895140864 _step‚263 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚182901138443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190514720771584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2845338584064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61131759 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42852411 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2845495917056 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61111722 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2709866403840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27371204 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187804854367744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205157871200768 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚190515395136512 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚2433.624516742 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187804854367744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15481654 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46921154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61111722 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚190514720771584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27371206 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2710522765824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2710522765824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42852860 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187804872370688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46921152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2709866403840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208003391057920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚3249 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61131759 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0‡é-qI2Ñ- _wandbruntime‚2434  _runtime‚2434>cºç&ÒáM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42865119 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205219545878528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46935170 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187861207406592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108085409 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚2434.297349716 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚182956042573824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61150239 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61130176 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚182956025534464 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61130176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108065344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚3249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15485747 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46935168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27379374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185656735939465 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190572567616512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884307495920568e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27379372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42865572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190571893238272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187861225409536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2700710405001 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2701364842961 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2711342206976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46935170 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187861207406592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2711342206976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2846367744000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚264 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190572567616512 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15486198 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187861225409536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2710685831680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177065472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205219521938432 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108065344 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚264 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190572567616512 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15486198 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187861225409536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2710685831680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177065472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205219521938432 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108065344 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42865119 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205219545878528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46935170 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187861207406592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108085409 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚2434.297349716 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚182956042573824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61150239 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61130176 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚182956025534464 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61130176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108065344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚3249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15485747 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46935168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27379374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185656735939465 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190572567616512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884307495920568e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27379372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42865572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190571893238272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187861225409536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2700710405001 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ˆu´Ú–2Ô- _wandbruntime‚2434  _runtime‚2434¿®xùUÕÏ /loss/FRt3/assignment_loss‚4.086902372215105 2loss/SRqqt2/assignment_loss‚1.7714685977057174 0loss/FBt3/assignment_loss‚0.6334755205763958 .loss/FRt2/detection_loss‚2.374658632417058 /loss/FRt3/detection_loss‚3.3112358528634775 /loss/FBt3/detection_loss‚2.9082545959032498 epoch‚2 /loss/FRt1/assignment_loss‚1.960289781869844 1loss/SRqqt4/assignment_loss‚1.596674857186336 .loss/FBt2/detection_loss‚2.598688671986262 $loss/total_loss‚48.7029009743687 2loss/SRqqt1/assignment_loss‚0.9170076321216597 .loss/FRt4/detection_loss‚2.307180335935281 1loss/SRqqt2/detection_loss‚1.8197632807272452 1loss/SRqqt4/detection_loss‚2.2768930827870086 0loss/FBt1/assignment_loss‚0.2999335614133018 1loss/SRqqt1/detection_loss‚1.2393960872314114 trainer/global_step‚3249 % _timestamp‚1.7884307495938265e+09 2loss/SRqqt3/assignment_loss‚2.3484222215154897 0loss/FBt2/assignment_loss‚0.1448674148026233 _step‚265 1loss/FBt4/assignment_loss‚0.33713003744681674 .loss/FRt1/detection_loss‚2.112718413475641 /loss/FBt4/detection_loss‚3.0871735711892447 _runtime‚2434.298450841 /loss/FRt4/assignment_loss‚4.179744664503604 /loss/FRt2/assignment_loss‚2.780759651993596 0loss/SRqqt3/detection_loss‚2.761915603409643 /loss/FBt1/detection_loss‚0.8483465330926941ÈOjZÖÔ 1loss/SRqqt4/detection_loss‚2.2768930827870086 0loss/FBt1/assignment_loss‚0.2999335614133018 1loss/SRqqt1/detection_loss‚1.2393960872314114 trainer/global_step‚3249 % _timestamp‚1.7884307495938265e+09 2loss/SRqqt3/assignment_loss‚2.3484222215154897 0loss/FBt2/assignment_loss‚0.1448674148026233 _step‚265 1loss/FBt4/assignment_loss‚0.33713003744681674 .loss/FRt1/detection_loss‚2.112718413475641 /loss/FBt4/detection_loss‚3.0871735711892447 _runtime‚2434.298450841 /loss/FRt4/assignment_loss‚4.179744664503604 /loss/FRt2/assignment_loss‚2.780759651993596 0loss/SRqqt3/detection_loss‚2.761915603409643 /loss/FBt1/detection_loss‚0.8483465330926941 /loss/FRt3/assignment_loss‚4.086902372215105 2loss/SRqqt2/assignment_loss‚1.7714685977057174 0loss/FBt3/assignment_loss‚0.6334755205763958 .loss/FRt2/detection_loss‚2.374658632417058 /loss/FRt3/detection_loss‚3.3112358528634775 /loss/FBt3/detection_loss‚2.9082545959032498 epoch‚2 /loss/FRt1/assignment_loss‚1.960289781869844 1loss/SRqqt4/assignment_loss‚1.596674857186336 .loss/FBt2/detection_loss‚2.598688671986262 $loss/total_loss‚48.7029009743687 2loss/SRqqt1/assignment_loss‚0.9170076321216597 .loss/FRt4/detection_loss‚2.307180335935281 1loss/SRqqt2/detection_loss‚1.8197632807272452‰DTéwׂ(:’ ¨“åÔÔ‡„Ž gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.91' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.581999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\è¡XØ‚(:— ¨“åÔóüµ‘$ memory_percent‚97.09994792938232 disk.rbd1.out‚ 1.2265625 disk.rbd0.in‚0 disk.rbd2.out‚ 90.10546875 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚0( proc.memory.availableMB‚ 475.14453125 cpu‚25.2892065361603 network.recv‚906212 disk.dm-1.in‚0 disk.sda2.in‚0 öoãÊdisk.sda3.out‚ 98.5234375 disk.sda1.out‚0# proc.memory.rssMB‚ 2870.19921875* proc.memory.percent‚17.518305778503418 disk.rbd4.out‚0 disk.nvme0n1.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda1.in‚0$ disk.nvme0n1.out‚12954.943359375 disk.sda.out‚ 98.5234375 proc.cpu.threads‚17 network.sent‚ 6.554707e+06% disk./.usageGB‚1100.2576751708984 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚12954.943359375 disk.sda3.in‚0 disk.drbd1777.out‚0! disk.dm-0.out‚12954.943359375) disk./.usagePercent‚73.83921575839835 disk.dm-1.out‚ 98.5234375 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd4.in‚06à›KžÙ‚(:“ ·“åÔî»ö gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.953' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.590600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Âä&Ú‚(:› ·“åÔ¤çÔ disk.rbd3.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 98.92578125 network.recv‚912450 disk.nvme0c0n1.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.dm-1.out‚ 98.92578125% disk.nvme0n1.out‚13054.3564453125 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.sda.out‚ 98.92578125 disk.sda1.out‚0' disk.nvme0c0n1.out‚13054.3564453125 disk.drbd1777.out‚0 disk.rbd4.out‚0" disk.dm-0.out‚13054.3564453125 disk.dm-1.in‚0 disk.sda2.out‚0 disk.rbd2.out‚ 90.10546875$ memory_percent‚97.09858894348145* proc.memory.percent‚17.518305778503418 cpu‚25.28777927087369 network.sent‚ 6.66349e+06) disk./.usagePercent‚73.83921447836069 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0$ disk./.usageGB‚1100.257656097412 disk.rbd1.out‚ 1.2265625# proc.memory.rssMB‚ 2870.19921875 disk.rbd0.out‚0' proc.memory.availableMB‚ 475.3671875 proc.cpu.threads‚17àã2Û- _wandbruntime‚2469  _runtime‚2469ÚÔ`”ÉÜà ' lr-AdamW/pg1‚0.00020661129727107325 ' lr-AdamW/pg2‚0.00020661129727107325 trainer/global_step‚3299 % _timestamp‚1.7884307846695654e+09 _runtime‚2469.374301968 _step‚266à/ÑÎÝÈ ' lr-AdamW/pg1‚0.00020661129727107325 ' lr-AdamW/pg2‚0.00020661129727107325 trainer/global_step‚3299 % _timestamp‚1.7884307846695654e+09 _runtime‚2469.374301968 _step‚266Š|ˆHÊ2Þ- _wandbruntime‚2469  _runtime‚2469.‰[Ý'ß×O OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43488491 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185646327931904 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190622506310144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2888386667008 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884307846704445e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47621952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188387057241349 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚267 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62037774 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190622506310144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2750852209664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2741383747197 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62037774 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109679765 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188387728718461 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚3299 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193373358519808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2469.375508192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47621954 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15708885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208240408083968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193373358519808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2750852209664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193374032884736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211128639515136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190622524313088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2751508571648 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211128818691072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109659726 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190622524313088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2740729309445Aåm´â'àÜO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62037774 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109679765 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188387728718461 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚3299 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193373358519808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2469.375508192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47621954 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15708885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208240408083968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193373358519808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2750852209664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193374032884736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211128639515136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190622524313088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2751508571648 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211128818691072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109659726 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190622524313088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2740729309445 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43488491 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185646327931904 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190622506310144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2888386667008 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884307846704445e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47621952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188387057241349 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚267 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62037774 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190622506310144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2750852209664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2741383747197 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844‹СÕ2á- _wandbruntime‚2470  _runtime‚2470U­m9%âáM E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2741545744409 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62076291 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211191151848960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208302058821632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2889254541312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27787772 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208302082761728 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211191337303040 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43500715 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚3299 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47635968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2752328012800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193431205364736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2751671637504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2751671637504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27787774 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2889093027328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190678859348992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193430530986496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2752328012800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62076291 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109712261 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109692196 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47635970 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185701232062464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188442777806873 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚268 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 çë;®JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Zߨuì&ãæM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47635968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2752328012800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193431205364736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2751671637504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2751671637504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27787774 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2889093027328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190678859348992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193430530986496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2752328012800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62076291 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109712261 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109692196 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47635970 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185701232062464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188442777806873 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚268 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2741545744409 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62076291 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211191151848960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208302058821632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2889254541312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27787772 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208302082761728 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211191337303040 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43500715 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚3299ŒL8~Û2ä- _wandbruntime‚2470  _runtime‚2470¬ÿj!WåÑ /loss/FRt4/detection_loss‚2.4616780033925685 1loss/SRqqt2/detection_loss‚1.8920019970220678 /loss/FBt2/detection_loss‚2.6604888985554376 % _timestamp‚1.7884307853287659e+09 0loss/FRt1/assignment_loss‚1.5737760650868318 2loss/SRqqt1/assignment_loss‚1.0779153528149987 1loss/FBt3/assignment_loss‚0.17881090318162762 /loss/FBt3/detection_loss‚3.9321547746658325 epoch‚2 1loss/SRqqt3/detection_loss‚2.4079876953647252 /loss/FRt3/assignment_loss‚3.988029144905709 1loss/SRqqt4/assignment_loss‚1.660443443942953 1loss/FBt1/assignment_loss‚0.15788183722388985 1loss/SRqqt1/detection_loss‚1.1348438906010057 .loss/FBt1/detection_loss‚0.704680109787084 1loss/SRqqt2/assignment_loss‚1.770063437229278 $loss/total_loss‚47.2978246737447 /loss/FRt4/assignment_loss‚4.128920979616119 0loss/FBt4/assignment_loss‚0.5438114367425442 1loss/SRqqt4/detection_loss‚1.8806551374770977 -loss/FBt4/detection_loss‚2.39224311709404 /loss/FRt1/detection_loss‚1.8515758711312498 1loss/SRqqt3/assignment_loss‚2.084459850830691 trainer/global_step‚3299 _runtime‚2470.033764535 _step‚269 0loss/FRt2/assignment_loss‚3.3930816501379013 0loss/FBt2/assignment_loss‚0.3457033784119024 .loss/FRt2/detection_loss‚2.347936507935325 /loss/FRt3/detection_loss‚2.7286811905938224u)Ú\æÖ 1loss/SRqqt2/assignment_loss‚1.770063437229278 $loss/total_loss‚47.2978246737447 /loss/FRt4/assignment_loss‚4.128920979616119 0loss/FBt4/assignment_loss‚0.5438114367425442 1loss/SRqqt4/detection_loss‚1.8806551374770977 -loss/FBt4/detection_loss‚2.39224311709404 /loss/FRt1/detection_loss‚1.8515758711312498 1loss/SRqqt3/assignment_loss‚2.084459850830691 trainer/global_step‚3299 _runtime‚2470.033764535 _step‚269 0loss/FRt2/assignment_loss‚3.3930816501379013 0loss/FBt2/assignment_loss‚0.3457033784119024 .loss/FRt2/detection_loss‚2.347936507935325 /loss/FRt3/detection_loss‚2.7286811905938224 /loss/FRt4/detection_loss‚2.4616780033925685 1loss/SRqqt2/detection_loss‚1.8920019970220678 /loss/FBt2/detection_loss‚2.6604888985554376 % _timestamp‚1.7884307853287659e+09 0loss/FRt1/assignment_loss‚1.5737760650868318 2loss/SRqqt1/assignment_loss‚1.0779153528149987 1loss/FBt3/assignment_loss‚0.17881090318162762 /loss/FBt3/detection_loss‚3.9321547746658325 epoch‚2 1loss/SRqqt3/detection_loss‚2.4079876953647252 /loss/FRt3/assignment_loss‚3.988029144905709 1loss/SRqqt4/assignment_loss‚1.660443443942953 1loss/FBt1/assignment_loss‚0.15788183722388985 1loss/SRqqt1/detection_loss‚1.1348438906010057 .loss/FBt1/detection_loss‚0.704680109787084Ê÷Øæžç‚(:“ Æ“åÔˆ…Ž gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.292' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.858400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x @+è‚(:  Æ“åÔø³¯ disk.sda.out‚ 99.35546875 network.recv‚916259 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 99.35546875 disk.rbd3.in‚0* proc.memory.percent‚17.518305778503418' disk.nvme0c0n1.out‚13153.3837890625 disk.sda2.in‚0 disk.rbd4.in‚0 disk.dm-1.out‚ 99.35546875# memory_percent‚97.1015214920044 cpu‚25.28635933329869) disk./.usagePercent‚73.83921575839835 disk.rbd0.in‚0 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd4.out‚0 proc.cpu.threads‚17 network.sent‚ 6.668732e+06% disk.nvme0n1.out‚13153.3837890625 disk.sda.in‚0( proc.memory.availableMB‚ 474.88671875 disk.rbd1.out‚ 1.2265625 disk.nvme0c0n1.in‚0 disk.rbd2.out‚90.21142578125 disk.drbd1777.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2870.19921875% disk./.usageGB‚1100.2576751708984" disk.dm-0.out‚13153.3837890625 disk.dm-1.in‚0Ü1ù“é‚(:ˆ Õ“åÔû„Ú gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08³­h'ê‚(:œ Õ“åÔÏÎÊ# proc.memory.rssMB‚ 2870.19921875 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚13252.5888671875 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 99.8671875 proc.cpu.threads‚17 network.sent‚ 6.777581e+06) disk./.usagePercent‚73.83921447836069 disk.dm-0.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0 disk.sda.out‚ 99.8671875$ disk./.usageGB‚1100.257656097412 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 99.8671875 disk.drbd1777.out‚0( proc.memory.availableMB‚ 475.07421875 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd2.out‚90.21142578125 disk.rbd4.in‚0" disk.dm-0.out‚13252.5888671875 disk.drbd1777.in‚0* proc.memory.percent‚17.518305778503418 disk.rbd1.out‚ 1.2265625 disk.rbd0.in‚0 cpu‚25.28494579687855' disk.nvme0c0n1.out‚13252.5888671875 disk.rbd0.out‚0 network.recv‚922803 disk.nvme0c0n1.in‚0# memory_percent‚97.1003770828247\Ðñžë‚(:“ ä“åÔLJŽ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚133.816' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.763199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ3ùí3ì‚(:¨ ä“åÔ¾ÝÖ‘ network.sent‚ 6.782999e+06 disk.sda2.out‚0 disk.drbd1777.out‚0 network.recv‚926755 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚13252.96728515625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚13252.96728515625 disk.rbd2.out‚90.28955078125$ memory_percent‚97.10042476654053( proc.memory.availableMB‚ 475.06640625 disk.sda.out‚ 100.25390625 disk.nvme0n1.in‚0) disk./.usagePercent‚73.83924161515931% disk./.usageGB‚1100.2580604553223 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 100.25390625 cpu‚25.283530759264593 proc.cpu.threads‚17 disk.rbd4.in‚0# disk.dm-0.out‚13252.96728515625 disk.rbd1.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.dm-1.out‚ 100.25390625 disk.rbd1.out‚ 1.2265625 disk.rbd0.out‚0 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2870.19921875* proc.memory.percent‚17.518305778503418¹@mž2í- _wandbruntime‚2505  _runtime‚2505Ef„ÇîÁ & lr-AdamW/pg2‚0.0002056733511819718 trainer/global_step‚3349 % _timestamp‚1.7884308206149356e+09 _runtime‚2505.319783583 _step‚270 & lr-AdamW/pg1‚0.0002056733511819718Ä ®ÌïÆ % _timestamp‚1.7884308206149356e+09 _runtime‚2505.319783583 _step‚270 & lr-AdamW/pg1‚0.0002056733511819718 & lr-AdamW/pg2‚0.0002056733511819718 trainer/global_step‚3349ŽFhžx2ð- _wandbruntime‚2505  _runtime‚2505™mPÝ'ñ×O YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚193440158252544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2792494377472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚2505.320736138 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48322752 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62963826 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111306617 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15937046 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44125494 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196232670632960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2931469448704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2782219086605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚193440176255488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚196231996268032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2781564648853 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚193440176255488 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196231996268032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚3349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28188006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 % _timestamp‚1.7884308206157398e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62983863 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2791838015488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44125050 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2791838015488 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48322754 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2931310018560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111286578 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214254438355968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188391551499264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62963826 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15937488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211322968907264 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2792494377472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚193440158252544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211322944967168 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚196232670632960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28188004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚214254254985728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188391534459904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414#ù.âòÜO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62963826 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15937488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211322968907264 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2792494377472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚193440158252544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211322944967168 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚196232670632960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28188004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚214254254985728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188391534459904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚193440158252544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2792494377472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚2505.320736138 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48322752 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62963826 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111306617 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15937046 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_røc‘=ejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44125494 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196232670632960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2931469448704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2782219086605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚193440176255488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚196231996268032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2781564648853 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚193440176255488 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196231996268032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚3349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28188006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 % _timestamp‚1.7884308206157398e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62983863 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2791838015488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44125050 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2791838015488 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48322754 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2931310018560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111286578 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214254438355968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188391551499264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872222ó- _wandbruntime‚2505  _runtime‚2505Ç«úQç&ôáM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48336768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214316955759616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884308212764645e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188446438590464 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2783035521777 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111339113 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211384595704832 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2792657443328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚193496529294336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63002343 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚196289843112960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196289168734720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15941616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188446455629824 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111339113 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44138233 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44137788 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28196172 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48336770 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63002343 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191228819674281 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15942059 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62982280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48336768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 _step‚272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193496511291392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2793313818624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0‰­Ñ¦ì&õæM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2932178795008 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63002343 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191228819674281 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15942059 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62982280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48336768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 _step‚272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193496511291392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2793313818624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48336768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214316955759616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884308212764645e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188446438590464 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2783035521777 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111339113 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211384595704832 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2792657443328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚193496529294336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63002343 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚196289843112960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196289168734720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15941616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188446455629824 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111339113 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44138233 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44137788 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28196172 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48336770 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217380037097984 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2822399988261 % _timestamp‚1.7884308560056722e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49023552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2833480183296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63909915 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49023554 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44762251 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193959812453277 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63889878 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214405481850368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193959140976165 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63909915 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2833480183296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28596406 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199091308381184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2832823821312 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44761801 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49023554 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199091308381184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217379849533440 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199090634016256 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196257810194944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63889878 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199090634016256 _step‚275 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191136758027264 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2823054426013 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16165397 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974367683072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191136740987904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2™â ˜â'‚ÜO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚3399 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217379849533440 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199090634016256 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196257810194944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63889878 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199090634016256 _step‚275 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191136758027264 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2823054426013 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16165397 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974367683072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191136740987904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚2540.710723878 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217380037097984 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2822399988261 % _timestamp‚1.7884308560056722e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49023552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2833480183296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63909915 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49023554 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44762251 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193959812453277 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63889878 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214405481850368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193959140976165 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63909915 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2833480183296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28596406 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199091308381184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2832823821312 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44761801 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49023554 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199091308381184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112965965 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28604574 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217442552876544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217442373713920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16170344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2833643249152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49037570 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63928395 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194014861541689 % _timestamp‚1.7884308566671793e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49037570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2975396348416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191191645118464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194015533019009 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63908332 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196314163233792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214467156528128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112945900 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2823216423225 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2975241125888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚191191662157824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚196314163233792 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16169914 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2823870861185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2834299624448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44774486 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚196314163233792 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_bÊ~µPÕatch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16169914 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2823870861185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2834299624448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44774486 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112965965 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28604574 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217442552876544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217442373713920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16170344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2833643249152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49037570 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63928395 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194014861541689 % _timestamp‚1.7884308566671793e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49037570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2975396348416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191191645118464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194015533019009 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63908332 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240”|Õ’2†- _wandbruntime‚2541  _runtime‚2541I•`pX‡Ò trainer/global_step‚3399 0loss/FRt1/assignment_loss‚1.8061995829145114 0loss/FBt1/assignment_loss‚0.2313633084386697 .loss/FRt3/detection_loss‚3.293101988732815 % _timestamp‚1.7884308566696765e+09 .loss/FBt4/assignment_loss‚0.57046599712755 1loss/SRqqt2/detection_loss‚1.7153066811608333 0loss/SRqqt3/detection_loss‚2.780775249004364 .loss/FBt2/detection_loss‚3.302415043115616 epoch‚2 2loss/SRqqt4/assignment_loss‚1.6756939432499083 .loss/FBt1/detection_loss‚0.559991087198796 2loss/SRqqt2/assignment_loss‚1.7959514981975742 1loss/SRqqt4/detection_loss‚2.1928345561027527 .loss/FBt3/detection_loss‚2.037557981230996 _step‚277 /loss/FRt4/assignment_loss‚3.890366369602727 2loss/SRqqt1/assignment_loss‚1.0667498651318825 1loss/FBt2/assignment_loss‚0.09264792682370171 /loss/FRt4/detection_loss‚2.1231531009954563 1loss/SRqqt1/detection_loss‚0.9375187932298734 /loss/FBt4/detection_loss‚2.0487785637378693 %loss/total_loss‚46.05812588071244 0loss/FRt2/assignment_loss‚3.3690732401244494 0loss/FRt3/assignment_loss‚3.6690732141335807 _runtime‚2541.374340587 2loss/SRqqt3/assignment_loss‚2.1591861069202425 0loss/FBt3/assignment_loss‚0.5125088942321864 /loss/FRt1/detection_loss‚1.9026000403488676 /loss/FRt2/detection_loss‚2.3248128489572175æÙ=Ý]ˆ× 0loss/FRt1/assignment_loss‚1.8061995829145114 0loss/FBt1/assignment_loss‚0.2313633084386697 .loss/FRt3/detection_loss‚3.293101988732815 % _timestamp‚1.7884308566696765e+09 .loss/FBt4/assignment_loss‚0.57046599712755 1loss/SRqqt2/detection_loss‚1.7153066811608333 0loss/SRqqt3/detection_loss‚2.780775249004364 .loss/FBt2/detection_loss‚3.302415043115616 epoch‚2 2loss/SRqqt4/assignment_loss‚1.6756939432499083 .loss/FBt1/detection_loss‚0.559991087198796 2loss/SRqqt2/assignment_loss‚1.7959514981975742 1loss/SRqqt4/detection_loss‚2.1928345561027527 .loss/FBt3/detection_loss‚2.037557981230996 _step‚277 /loss/FRt4/assignment_loss‚3.890366369602727 2loss/SRqqt1/assignment_loss‚1.0667498651318825 1loss/FBt2/assignment_loss‚0.09264792682370171 /loss/FRt4/detection_loss‚2.1231531009954563 1loss/SRqqt1/detection_loss‚0.9375187932298734 /loss/FBt4/detection_loss‚2.0487785637378693 %loss/total_loss‚46.05812588071244 0loss/FRt2/assignment_loss‚3.3690732401244494 0loss/FRt3/assignment_loss‚3.6690732141335807 _runtime‚2541.374340587 2loss/SRqqt3/assignment_loss‚2.1591861069202425 0loss/FBt3/assignment_loss‚0.5125088942321864 /loss/FRt1/detection_loss‚1.9026000403488676 /loss/FRt2/detection_loss‚2.3248128489572175 trainer/global_step‚3399•  ‰‚(:’ ‘”åÔû¬ Ž gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.566000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*¨±"Š‚(:— ‘”åÔÃÛ— disk.dm-1.in‚0 disk.dm-1.out‚ 101.828125 disk.rbd0.out‚0 disk.sda3.out‚ 101.828125 disk.sda.out‚ 101.828125 cpu‚25.27948154664444) disk./.usagePercent‚73.83684026448815 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd4.out‚0$ memory_percent‚97.10078239440918 proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.rbd1.out‚1.3125 disk.rbd4.in‚0' proc.memory.availableMB‚ 475.0078125* proc.memory.percent‚17.518305778503418$ disk.nvme0n1.out‚13551.642578125 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0 network.sent‚ 6.902532e+06 network.recv‚940436 disk.nvme0c0n1.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2870.19921875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚13551.642578125 disk.sda2.out‚0 disk.rbd2.out‚91.94189453125 disk.dm-0.in‚0! disk.dm-0.out‚13551.642578125% disk./.usageGB‚1100.2222785949707ðñ·µ‘‹‚(:†  ”åÔß¿å gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚98.896' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.7792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß^h*Œ‚(:Ÿ  ”åÔ¤¬ disk.rbd1.out‚1.375' disk.nvme0c0n1.out‚13551.9501953125# proc.memory.rssMB‚ 2870.19921875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.nvme0n1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.518305778503418 disk.sda2.out‚0 network.sent‚ 7.010516e+06) disk./.usagePercent‚73.83684154452581 network.recv‚946222 disk.sda2.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0( proc.memory.availableMB‚ 475.00390625" disk.dm-0.out‚13551.9501953125 disk.dm-1.out‚ 102.30078125% disk.nvme0n1.out‚13551.9501953125 disk.sda3.out‚ 102.30078125 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0$ memory_percent‚97.10080623626709 cpu‚25.278149828698197 disk.rbd2.out‚ 92.1142578125 disk.sda.out‚ 102.30078125 disk.sda3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚0$ disk./.usageGB‚1100.222297668457 disk.rbd1.in‚0e¸V2- _wandbruntime‚2576  _runtime‚2576ét¹ÕÈŽÂ trainer/global_step‚3449 $ _timestamp‚1.788430891449235e+09 _runtime‚2576.153951966 _step‚278 ' lr-AdamW/pg1‚0.00020373991420712776 ' lr-AdamW/pg2‚0.00020373991420712776@\ÀÍÇ trainer/global_step‚3449 $ _timestamp‚1.788430891449235e+09 _runtime‚2576.153951966 _step‚278 ' lr-AdamW/pg1‚0.00020373991420712776 ' lr-AdamW/pg2‚0.00020373991420712776–ùMK°2- _wandbruntime‚2576  _runtime‚25762™ Ý'‘×O L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49724352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201949271764480 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201949946129408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201949271764480 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199075462137344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29004804 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚193881947515904 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64815930 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49724352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45397612 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3017320181248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚196745182843573 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2863889765421 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199075480140288 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220505515993600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64835967 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49724354 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201949946129408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2873809627136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64815930 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2874465989120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217488042673664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2873809627136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220505338914816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3017473319936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201949271764480 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201949946129408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201949271764480 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199075462137344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29004804 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚193881947515904 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64815930 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49724352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45397612 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3017320181248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚196745182843573 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2863889765421 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199075480140288 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220505515993600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64835967 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49724354 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201949946129408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2873809627136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64815930 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2874465989120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217488042673664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2873809627136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22050533891481ã§EP6 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3017473319936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217549693411328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199131833179136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45410304 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114572752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64854447 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚196801574886417 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚196800903409097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29012972 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202006444231168 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45409871 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚280 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2864051762633 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202007118609408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64834384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217549669471232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114592817 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64854447 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202007118609408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844ò§üÍì&•æM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202007118609408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64834384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217549669471232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114592817 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64854447 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202007118609408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217549693411328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199131833179136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45410304 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114572752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64854447 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚196801574886417 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚196800903409097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29012972 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202006444231168 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45409871 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚280 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2864051762633 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414˜£y%2–- _wandbruntime‚2576  _runtime‚2576mûR›[—Õ /loss/FRt1/detection_loss‚1.6482637600174972 2loss/SRqqt1/assignment_loss‚1.1039678395088566 /loss/FBt4/detection_loss‚2.8206755585140653 epoch‚2 % _timestamp‚1.7884308921018004e+09 /loss/FRt3/detection_loss‚3.0420384407043457 1loss/SRqqt2/assignment_loss‚1.632937375551615 .loss/FBt2/detection_loss‚2.719155212243398 trainer/global_step‚3449 _runtime‚2576.806465973 0loss/FRt2/assignment_loss‚3.2115996441115504 /loss/FRt3/assignment_loss‚4.010168855006878 2loss/SRqqt4/assignment_loss‚1.8258063789795747 0loss/FBt3/assignment_loss‚0.4711321696639061 .loss/FRt4/detection_loss‚2.489639188898237 &loss/total_loss‚47.881985535547095 .loss/FRt2/detection_loss‚2.321858786370443 /loss/FRt4/assignment_loss‚4.159002188004945 1loss/FBt1/assignment_loss‚0.25459357687007017 1loss/FBt2/assignment_loss‚0.15630321891399035 0loss/SRqqt2/detection_loss‚2.076262715535286 /loss/FBt1/detection_loss‚0.6455351897097917 /loss/FBt3/detection_loss‚2.9202263355255127 _step‚281 /loss/FRt1/assignment_loss‚1.809501051902771 2loss/SRqqt3/assignment_loss‚2.3682774616913362 1loss/FBt4/assignment_loss‚0.31556736553708714 1loss/SRqqt1/detection_loss‚0.9317313273878474 1loss/SRqqt3/detection_loss‚2.8730292157693342 1loss/SRqqt4/detection_loss‚2.0747126791287553d¼$`˜Ú trainer/global_step‚3449 _runtime‚2576.806465973 0loss/FRt2/assignment_loss‚3.2115996441115504 /loss/FRt3/assignment_loss‚4.010168855006878 2loss/SRqqt4/assignment_loss‚1.8258063789795747 0loss/FBt3/assignment_loss‚0.4711321696639061 .loss/FRt4/detection_loss‚2.489639188898237 &loss/total_loss‚47.881985535547095 .loss/FRt2/detection_loss‚2.321858786370443 /loss/FRt4/assignment_loss‚4.159002188004945 1loss/FBt1/assignment_loss‚0.25459357687007017 1loss/FBt2/assignment_loss‚0.15630321891399035 0loss/SRqqt2/detection_loss‚2.076262715535286 /loss/FBt1/detection_loss‚0.6455351897097917 /loss/FBt3/detection_loss‚2.9202263355255127 _step‚281 /loss/FRt1/assignment_loss‚1.809501051902771 2loss/SRqqt3/assignment_loss‚2.3682774616913362 1loss/FBt4/assignment_loss‚0.31556736553708714 1loss/SRqqt1/detection_loss‚0.9317313273878474 1loss/SRqqt3/detection_loss‚2.8730292157693342 1loss/SRqqt4/detection_loss‚2.0747126791287553 /loss/FRt1/detection_loss‚1.6482637600174972 2loss/SRqqt1/assignment_loss‚1.1039678395088566 /loss/FBt4/detection_loss‚2.8206755585140653 epoch‚2 % _timestamp‚1.7884308921018004e+09 /loss/FRt3/detection_loss‚3.0420384407043457 1loss/SRqqt2/assignment_loss‚1.632937375551615 .loss/FBt2/detection_loss‚2.719155212243398™Ï5ëž™‚(:“ ¯”åÔƒ¦è gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.272' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.454399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô”Ý +š‚(:  ¯”åÔéÝœ‘ disk.sda.out‚ 102.98828125 disk.nvme0n1.in‚0* proc.memory.percent‚17.518305778503418 cpu‚25.276723068311096 proc.cpu.threads‚17 disk.rbd2.out‚ 92.1142578125 disk.drbd1777.in‚0" disk.dm-0.out‚13655.0166015625( proc.memory.availableMB‚ 474.98046875 disk.sda3.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚97.10094928741455 disk.sda3.out‚ 102.98828125 disk.rbd1.in‚0 disk.rbd1.out‚1.375# proc.memory.rssMB‚ 2870.19921875 network.sent‚ 7.016783e+06 disk.nvme0c0n1.in‚0 disk.sda.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚13655.0166015625 disk.rbd0.out‚0) disk./.usagePercent‚73.83704788659841% disk./.usageGB‚1100.2253723144531 disk.sda2.in‚0 network.recv‚950484 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 102.98828125' disk.nvme0c0n1.out‚13655.0166015625 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.dm-0.in‚0ºNDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46033415 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223631233164288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚201893132082688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29413206 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚204808583877632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚223631045599744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2904725104829 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚201893132082688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16620211 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884309269044719e+09 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚204807909512704 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚204807909512704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50425154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29413204 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚283 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚204808583877632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2914795432960ÑÄ×iâ'¤ÜO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50425154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29413204 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚283 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚204808583877632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2914795432960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚199531896188093 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46033415 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223631233164288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚201893132082688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29413206 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚204808583877632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚223631045599744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2904725104829 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚201893132082688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16620211 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884309269044719e+09 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚204807909512704 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚204807909512704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0›º(ˆÏ2¥- _wandbruntime‚2612  _runtime‚2612ÌJÚç&¦áM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29421374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚2612.269318428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29421372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚284 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚201949467118592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220632230294528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2915614860800 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚204865081979392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2916271236096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2905541540001 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2915614860800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16624657 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2916271236096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199587616753825 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199586945276505 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16625104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3061517095936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116219669 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚201949467118592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223693747390464 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116199604 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65760436 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116219669 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46046029 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65760436 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50439170 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46046478 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196682075213824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50439168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220632206354432 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 õú(ø§æM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2904887102041 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 trainer/global_step‚3499 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚204865081979392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2916271236096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2905541540001 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2915614860800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16624657 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2916271236096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199587616753825 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199586945276505 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16625104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3061517095936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116219669 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚201949467118592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223693747390464 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116199604 FDeviceS• «ôtatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65760436 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116219669 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46046029 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65760436 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50439170 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46046478 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196682075213824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50439168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220632206354432 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29421374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚2612.269318428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29421372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚284 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚201949467118592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220632230294528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2915614860800 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207666547260928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51125954 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2956437600768 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚223653092499968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199372360571904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚223653116440064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226756671397888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117793986 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46670100 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46669646 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204710784025088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226756863156736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2945560444237 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16848042 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66688071 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204710766022144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207666547260928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844ïÓ!â'´ÜO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66688071 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204710766022144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207666547260928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207666547260928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51125954 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2956437600768 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚223653092499968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199372360571904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚223653116440064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226756671397888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117793986 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46670100 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46669646 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204710784025088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226756863156736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2945560444237 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16848042Ÿ7£ƒü2µ- _wandbruntime‚2647  _runtime‚2647ç}3¶áM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51139970 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3104419685376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2647.714838674 _step‚288 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204767137063936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117846521 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚223714743237632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117826456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51139968 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207723719727616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3104597976576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JEÃÝÆ´@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204767119060992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2956600666624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202373658621233 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117826456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51139970 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226819365154304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202372987143913 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204767119060992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46682790 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231296 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199427264702464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2945722441449 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2957257041920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66706551 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207724394105856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 trainer/global_step‚3549 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204767137063936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66686488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29829774 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884309630096538e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16853016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16852555 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29829772 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226819162923008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46682327 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207724394105856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51139968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66686488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2946376879409 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117846521 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2956600666624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚223714767177728 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66706551 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199427281741824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844cîì&·æM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117826456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51139970 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226819365154304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202372987143913 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204767119060992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46682790 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231296 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199427264702464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2945722441449 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2957257041920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66706551 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207724394105856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 trainer/global_step‚3549 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204767137063936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66686488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29829774 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884309630096538e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16853016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16852555 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29829772 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226819162923008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46682327 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207724394105856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51139968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66686488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2946376879409 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117846521 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2956600666624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚223714767177728 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66706551 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199427281741824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51139970 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3104419685376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2647.714838674 _step‚288 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204767137063936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117846521 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚223714743237632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117826456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51139968 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207723719727616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3104597976576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204767119060992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2956600666624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202373658621233 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 µWÏ‚2¸- _wandbruntime‚2647  _runtime‚2647¡¾óõU¹Ï /loss/FBt4/detection_loss‚3.8280336260795593 1loss/SRqqt3/assignment_loss‚2.198624114609427 1loss/FBt3/assignment_loss‚0.16923466786101926 0loss/FBt4/assignment_loss‚0.3022151403129101 0loss/FRt3/assignment_loss‚3.5554405533034226 /loss/FBt1/assignment_loss‚0.218381291696668 1loss/SRqqt2/detection_loss‚1.8942327239999064 /loss/FBt1/detection_loss‚0.5753839641890636 /loss/FRt2/assignment_loss‚3.038557619518704 /loss/FRt4/detection_loss‚2.6105393510918287 1loss/SRqqt4/detection_loss‚1.9080660413293278 trainer/global_step‚3549 1loss/SRqqt2/assignment_loss‚1.802139490014977 .loss/FRt3/detection_loss‚2.937537513930222 _step‚289 2loss/SRqqt4/assignment_loss‚1.4751469900502878 _runtime‚2647.715837096 /loss/FRt1/assignment_loss‚1.884713523917728 .loss/FRt1/detection_loss‚1.993058397072471 .loss/FRt2/detection_loss‚2.261363336775038 1loss/SRqqt3/detection_loss‚2.2105217195219464 .loss/FBt3/detection_loss‚4.056291937828064 /loss/FRt4/assignment_loss‚3.779304587563803 0loss/SRqqt1/detection_loss‚1.023973910565916 &loss/total_loss‚47.865456713381356 epoch‚2 % _timestamp‚1.7884309630111964e+09 2loss/SRqqt1/assignment_loss‚0.9864338739095507 0loss/FBt2/assignment_loss‚0.8626507711107313 .loss/FBt2/detection_loss‚2.293611567128788Ùê(øZºÔ /loss/FRt1/assignment_loss‚1.884713523917728 .loss/FRt1/detection_loss‚1.993058397072471 .loss/FRt2/detection_loss‚2.261363336775038 1loss/SRqqt3/detection_loss‚2.2105217195219464 .loss/FBt3/detection_loss‚4.056291937828064 /loss/FRt4/assignment_loss‚3.779304587563803 0loss/SRqqt1/detection_loss‚1.023973910565916 &loss/total_loss‚47.865456713381356 epoch‚2 % _timestamp‚1.7884309630111964e+09 2loss/SRqqt1/assignment_loss‚0.9864338739095507 0loss/FBt2/assignment_loss‚0.8626507711107313 .loss/FBt2/detection_loss‚2.293611567128788 /loss/FBt4/detection_loss‚3.8280336260795593 1loss/SRqqt3/assignment_loss‚2.198624114609427 1loss/FBt3/assignment_loss‚0.16923466786101926 0loss/FBt4/assignment_loss‚0.3022151403129101 0loss/FRt3/assignment_loss‚3.5554405533034226 /loss/FBt1/assignment_loss‚0.218381291696668 1loss/SRqqt2/detection_loss‚1.8942327239999064 /loss/FBt1/detection_loss‚0.5753839641890636 /loss/FRt2/assignment_loss‚3.038557619518704 /loss/FRt4/detection_loss‚2.6105393510918287 1loss/SRqqt4/detection_loss‚1.9080660413293278 trainer/global_step‚3549 1loss/SRqqt2/assignment_loss‚1.802139490014977 .loss/FRt3/detection_loss‚2.937537513930222 _step‚289 2loss/SRqqt4/assignment_loss‚1.4751469900502878 _runtime‚2647.715837096¡õ»‚(:’ ú”åÔÂã gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚92.144' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.428800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~:ö.¼‚(:£ ú”åԦܕ‘% disk./.usageGB‚1100.2254676818848 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.sda.out‚ 105.671875( disk.nvme0c0n1.out‚14052.16357421875* proc.memory.percent‚17.518305778503418 network.recv‚975608 disk.rbd2.in‚0 disk.dm-1.out‚ 105.671875& disk.nvme0n1.out‚14052.16357421875 disk.rbd1.in‚0# memory_percent‚97.1017599105835 proc.cpu.threads‚17) disk./.usagePercent‚73.83705428678677 disk.drbd1777.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 1.533203125 disk.nvme0c0n1.in‚0( proc.memory.availableMB‚ 474.84765625 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda2.out‚0 cpu‚25.270288419070752 disk.sda1.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0 network.sent‚ 7.250737e+06 disk.sda3.out‚ 105.671875# disk.dm-0.out‚14052.16357421875 disk.rbd4.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚94.00146484375# proc.memory.rssMB‚ 2870.19921875tÂG8’½‚(:‡ ‰•åÔË·ã gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚94.861' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð¶Ï` ¾‚(:• ‰•åÔݱö disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.out‚ 14151.5234375' proc.memory.availableMB‚ 474.9921875 cpu‚25.26908687399771 disk.sda.in‚0 disk.nvme0n1.in‚0 disk.rbd2.out‚94.01708984375 disk.dm-0.in‚0 proc.cpu.threads‚17 network.sent‚ 7.359991e+06$ disk./.usageGB‚1100.225341796875 disk.dm-1.out‚ 106.203125 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd4.out‚0# proc.memory.rssMB‚ 2870.19921875 disk.sda.out‚ 106.203125 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.518305778503418) disk./.usagePercent‚73.83704583853815 disk.dm-1.in‚0 disk.rbd1.out‚ 1.533203125$ disk.nvme0c0n1.out‚ 14151.5234375 disk.sda3.out‚ 106.203125 disk.rbd4.in‚0 network.recv‚982348" disk.nvme0n1.out‚ 14151.5234375 disk.rbd2.in‚0 disk.drbd1777.in‚0$ memory_percent‚97.10087776184082ðûX2¿- _wandbruntime‚2682  _runtime‚2682¡_1VÈÀ trainer/global_step‚3599 $ _timestamp‚1.788430997980519e+09 _runtime‚2682.685273713 _step‚290 ' lr-AdamW/pg1‚0.00020069894821107798 ' lr-AdamW/pg2‚0.00020069894821107798…÷YÍÁÇ ' lr-AdamW/pg1‚0.00020069894821107798 ' lr-AdamW/pg2‚0.00020069894821107798 trainer/global_step‚3599 $ _timestamp‚1.788430997980519e+09 _runtime‚2682.685273713 _step‚290¢lj[2Â- _wandbruntime‚2682  _runtime‚2682‹óæû§!Ã×O QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47306521 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚205103308445797 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚229882402313728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚229882210554880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226735629383168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67614123 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2985741345893 trainer/global_step‚3599 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17076068 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202117584139264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚210525185009152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚210525859374080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatÁ¡/6sMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51826752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17076515 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2985741345893 trainer/global_step‚3599 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17076068 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202117584139264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚210525185009152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚210525859374080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51826752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17076515 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67614123 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517£G×`S2Å- _wandbruntime‚2683  _runtime‚2683ÖnG-ç&ÆáM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚3599 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119453308 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51840770 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67632603 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚2683.348814155 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51840768 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚210583031854080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207584789006336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51840768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202172471230464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67632603 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2997586472448 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884309986434448e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚205159029011321 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚292 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67612540 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2998242847744 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3147448773632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2987212218817 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229944728894464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202172488269824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119473373 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3147608190464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47319180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚226797280120832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚205159700488641 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51840770 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚210582357475840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207584789006336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚210583031854080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119473373 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47318741 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚226797304060928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2997586472448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30238172 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 Eýì&ÇæM FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51840768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202172471230464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67632603 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2997586472448 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884309986434448e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚205159029011321 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚292 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67612540 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2998242847744 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3147448773632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2987212218817 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229944728894464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202172488269824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119473373 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3147608190464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47319180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚226797280120832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚205159700488641 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51840770 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚210582357475840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207584789006336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚210583031854080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119473373 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47318741 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚226797304060928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2997586472448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30238172 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51840770 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67632603 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚2683.348814155 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51840768 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚210583031854080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207584789006336¤Å#,-2È- _wandbruntime‚2683  _runtime‚2683@0É–€ÉÌ /loss/FRt2/assignment_loss‚2.854166569917098 2loss/SRqqt2/assignment_loss‚1.7140896985697192 0loss/SRqqt3/detection_loss‚2.452844990738507 epoch‚2 1loss/FBt1/assignment_loss‚0.19779878176299767 _runtime‚ 2683.34981938 /loss/FRt1/assignment_loss‚1.984676394197676 1loss/SRqqt4/detection_loss‚1.6200654117428526 % _timestamp‚1.7884309986451848e+09 /loss/FRt2/detection_loss‚2.4410463610421056 $loss/total_loss‚46.4299300387075 /loss/FRt3/assignment_loss‚4.328823085342135 0loss/FBt2/assignment_loss‚0.2650665312534102 .loss/FRt1/detection_loss‚2.024840029908551 trainer/global_step‚3599 _step‚293 /loss/FRt4/assignment_loss‚3.602112809816996 2loss/SRqqt1/assignment_loss‚0.7609240810658472 .loss/FRt3/detection_loss‚3.001402016196932 /loss/FRt4/detection_loss‚2.1619337187872993 1loss/SRqqt1/detection_loss‚bÂÖÒ1.4835787946358323 /loss/FBt1/detection_loss‚0.5727887612916048 .loss/FBt2/detection_loss‚2.999787300825119 2loss/SRqqt4/assignment_loss‚1.7327017643469937 1loss/SRqqt2/detection_loss‚1.9891228862973147 -loss/FBt3/detection_loss‚3.26917349208485 .loss/FBt4/detection_loss‚2.165554968516032 1loss/SRqqt3/assignment_loss‚1.987632795140661 0loss/FBt3/assignment_loss‚0.2622349038720118 0loss/FBt4/assignment_loss‚0.5575638913549483‡àWÊÑ 0loss/FBt3/assignment_loss‚0.2622349038720118 0loss/FBt4/assignment_loss‚0.5575638913549483 /loss/FRt2/assignment_loss‚2.854166569917098 2loss/SRqqt2/assignment_loss‚1.7140896985697192 0loss/SRqqt3/detection_loss‚2.452844990738507 epoch‚2 1loss/FBt1/assignment_loss‚0.19779878176299767 _runtime‚ 2683.34981938 /loss/FRt1/assignment_loss‚1.984676394197676 1loss/SRqqt4/detection_loss‚1.6200654117428526 % _timestamp‚1.7884309986451848e+09 /loss/FRt2/detection_loss‚2.4410463610421056 $loss/total_loss‚46.4299300387075 /loss/FRt3/assignment_loss‚4.328823085342135 0loss/FBt2/assignment_loss‚0.2650665312534102 .loss/FRt1/detection_loss‚2.024840029908551 trainer/global_step‚3599 _step‚293 /loss/FRt4/assignment_loss‚3.602112809816996 2loss/SRqqt1/assignment_loss‚0.7609240810658472 .loss/FRt3/detection_loss‚3.001402016196932 /loss/FRt4/detection_loss‚2.1619337187872993 1loss/SRqqt1/detection_loss‚1.4835787946358323 /loss/FBt1/detection_loss‚0.5727887612916048 .loss/FBt2/detection_loss‚2.999787300825119 2loss/SRqqt4/assignment_loss‚1.7327017643469937 1loss/SRqqt2/detection_loss‚1.9891228862973147 -loss/FBt3/detection_loss‚3.26917349208485 .loss/FBt4/detection_loss‚2.165554968516032 1loss/SRqqt3/assignment_loss‚1.987632795140661¥ÏVضžË‚(:“ ˜•åÔ¦éŽ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.547' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.109399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šÇN7(Ì‚(: ˜•åÔ„¥ò disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.sda.out‚ 106.71484375 disk.sda3.in‚0& proc.memory.availableMB‚ 474.328125 cpu‚25.26776366876773 network.recv‚986556 disk.rbd2.in‚0 disk.sda3.out‚ 106.71484375 disk.sda1.in‚0 disk.nvme0n1.in‚0 disk.rbd0.in‚0$ memory_percent‚97.10493087768555# proc.memory.rssMB‚ 2870.19921875* proc.memory.percent‚17.518305778503418 proc.cpu.threads‚17 network.sent‚ 7.36558e+06 disk.rbd4.out‚0 disk.dm-0.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 disk.drbd1777.in‚0) disk./.usagePercent‚73.83711854467786 disk.sda1.out‚0% disk.nvme0c0n1.out‚14250.59765625 disk.drbd1777.out‚0 disk.dm-0.out‚14250.59765625# disk.nvme0n1.out‚14250.59765625 disk.rbd1.out‚ 1.533203125 disk.rbd2.out‚95.59130859375 disk.dm-1.out‚ 106.71484375 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0% disk./.usageGB‚1100.2264251708984Ï­]““Í‚(:ˆ §•åÔÿåù gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.0802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tN2΂(:§ §•åÔ¡…É) disk./.usagePercent‚73.83711521657992 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd4.out‚0# disk.dm-0.out‚14349.80908203125 disk.sda.in‚0$ disk./.usageGB‚1100.226375579834 disk.rbd1.out‚ 1.533203125 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚97.10278511047363* proc.memory.percent‚17.518305778503418& disk.nvme0n1.out‚14349.80908203125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 107.16796875 proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚14349.80908203125 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.dm-0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 107.16796875 cpu‚25.26666329253231 network.sent‚ 7.474316e+06 disk.sda2.out‚0 disk.rbd2.out‚95.60302734375 disk.sda3.out‚ 107.16796875' proc.memory.availableMB‚ 474.6796875# proc.memory.rssMB‚ 2870.19921875 network.recv‚992996!4±.œÏ‚(:‘ ¶•åÔ½âþ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚86.859' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.371799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ}÷\(Ђ(: ¶•åÔȼ¦‘ cpu‚25.288904060091742 disk.sda3.out‚ 107.53125( proc.memory.availableMB‚ 547.16015625 network.sent‚ 7.479592e+06( disk./.usagePercent‚73.8371203367306 disk.rbd4.out‚0 disk.dm-0.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚996843 disk.sda.out‚ 107.53125 disk.nvme0c0n1.in‚0$ memory_percent‚96.66039943695068 disk.sda.in‚0 disk.dm-1.out‚ 107.53125 disk.nvme0n1.in‚0 disk.rbd1.out‚ 1.533203125 disk.rbd2.out‚95.72802734375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚14350.13623046875 disk.sda2.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚14350.13623046875 disk.sda1.out‚0 disk.rbd0.in‚0% disk./.usageGB‚1100.2264518737793 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚14350.13623046875 disk.sda2.out‚0" proc.memory.rssMB‚ 2797.9609375( proc.memory.percent‚17.0773983001709H@ñOžÑ‚(:“ Å•åÔÞÉ› gpu.0.gpu‚58.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚161.427' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.285399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èãk4(Ò‚(: Å•åÔ ¢Ú–( disk./.usagePercent‚73.8371188006854 disk.sda1.in‚0 disk.sda3.out‚ 108.171875 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.533203125( proc.memory.availableMB‚ 474.05078125 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 108.171875 disk.sda2.out‚0 disk.rbd0.out‚0$ memory_percent‚97.10662364959717 disk.sda1.out‚0 disk.nvme0c0n1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2870.62109375 proc.cpu.threads‚17% disk./.usageGB‚1100.2264289855957 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.rbd2.out‚95.74365234375 disk.dm-0.out‚ 14449.6328125 disk.dm-1.in‚0 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 14449.6328125* proc.memory.percent‚17.520880699157715 cpu‚25.314350240526704 network.recv‚ 1.001622e+06 disk.rbd2.in‚0 network.sent‚ 7.488729e+06 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.sda.out‚ 108.171875" disk.nvme0n1.out‚ 14449.6328125 disk.sda2.in‚0À7Š‘Ó‚(:† Ô•åÔå¦ú gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚86.462' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.2924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü’·÷2Ô‚(:§ Ô•åÔÑʾ‘ disk.drbd1777.out‚0" disk.dm-0.out‚14548.7431640625 disk.nvme0c0n1.in‚0 disk.sda3.out‚ 108.57421875( proc.memory.percent‚17.0773983001709 disk.dm-1.out‚ 108.57421875 disk.rbd1.out‚ 1.533203125 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd4.in‚0$ memory_percent‚96.66135311126709" proc.memory.rssMB‚ 2797.9609375 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚0( proc.memory.availableMB‚ 547.00390625( disk./.usagePercent‚73.8371188006854 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 cpu‚25.340536516501214 disk.sda.out‚ 108.57421875' disk.nvme0c0n1.out‚14548.7431640625 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 7.491988e+06% disk./.usageGB‚1100.2264289855957 disk.sda.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚14548.7431640625 disk.rbd2.in‚0 disk.rbd2.out‚ 95.892578125 disk.sda3.in‚0 network.recv‚ 1.005434e+06 disk.sda1.in‚0 disk.nvme0n1.in‚0,Kñ2Õ- _wandbruntime‚2749  _runtime‚2749K›DND*Ö¾T YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3037785559040 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 3038278388736 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15976 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚210910992466432 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 121221824 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚213949270855168 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 134218752 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3188828548608 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚30777404 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚17419938 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953748 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15978 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚210910974463488 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955585536 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚210910992466432 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚52748696 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955585536 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492829696 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853517hR” I*×ÃT _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 3037785559040 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24400 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 3026602812437 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚210910974463488 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15978 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 121221824 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚213949270855168 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 134218752 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3188828548608 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚30777404 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚17419938 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953748 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15978 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚210910974463488 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955585536 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚210910992466432 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚52748696 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955585536 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492829696 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853517 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3037785559040 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 3038278388736 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15976 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚210910992466432 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_end/oversize_seg–°¹·"ments.allocated‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953748 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914388 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚68496408 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 3038643515392 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829696 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚52759500 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 3189193675264 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884310659541483e+09 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24127 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 _runtime‚2750.659284643 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚233544988547072 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 3038643515392 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚68496408 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15976 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815168 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚30783908 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚230355636712960 trainer/global_step‚3626 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953748 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914388 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚68496408 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 3038643515392 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829696 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚52759500 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 3189193675264 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884310659541483e+09 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24127 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 _runtime‚2750.659284643 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚233544988547072 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 3038643515392 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚68496408 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15976 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815168 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚30783908 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚230355636712960 trainer/global_step‚3626 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 B&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25295308232307434 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5040625333786011 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.058811720460653305 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4896259307861328 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.38930708169937134 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 $ _timestamp‚1.788431078757952e+09 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9893487691879272 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3723905086517334 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5578149557113647 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21281607449054718 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.983271598815918 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.380768746137619 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.44460251927375793 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48325392603874207 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.455828994512558 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4857316017150879 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40435731410980225 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4510582387447357 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.3719954490661621 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0476190485060215 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9809584617614746 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.48576676845550537 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4766666889190674 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4282742440700531 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 epoch‚2 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt1SRqqt*FBt/even Üqª±et_proportion‚0.2310267835855484 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21729157865047455 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9834064841270447 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4640178680419922 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16003088653087616 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9812644720077515 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1460905373096466 jet/accuracy_4_of_6‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15064555406570435 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5333333611488342 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121212482452393 jet/accuracy_4_of_7‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.41662806272506714 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9863973259925842 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3703703284263611 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7341541051864624 particle/accuracy_2_of_4‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36666664481163025 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5526316165924072 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22574958205223083 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9857913851737976 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3512346148490906 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.022012578323483467 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0778001993894577 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01455671526491642 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 +jet/accuracy_4_of_4‚0.00365606602281332 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36423614621162415 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 jet/accuracy_1_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00382487871684134 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15441717207431793 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21064814925193787 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08525852113962173 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.41542014479637146 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3539038598537445 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12817524373531342 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.03125 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22033430635929108 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9865415692329407 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15486213564872742 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4252593517303467 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 particle/accuracy_1_of_5‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4313177764415741 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8859649300575256 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.025486430153250694 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4246675372123718 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06592261791229248 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.23254819214344025 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.48364463448524475 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9869614839553833 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3672953248023987 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 +jet/accuracy_2_of_4‚0.35793372988700867 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3024691939353943 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952027797698975 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17207247018814087 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4580737054347992 particle/accuracy_3_of_5‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24814818799495697 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 particle/accuracy_5_of_5‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 +jet/accuracy_3_of_6‚0.07692307978868484 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6289891004562378 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3583333194255829 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2358071506023407 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16934022307395935 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9902847409248352 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04813218116760254 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21724137663841248 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24852797389030457 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 _step‚296 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12807594239711761 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20189380645751953 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4841269552707672 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4424987733364105 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12222221493721008 particle/accuracy_2_of_3‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5128205418586731 jet/accuracy_6_of_7‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5361305475234985 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9437602758407593 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.40838780999183655 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5624999403953552 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996392130851746 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3766271471977234 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13432864844799042 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20954936742782593 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4544389843940735 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,particle/sensitivity‚0.29808613657951355 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.98689866065979 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 particle/accuracy_4_of_7‚1 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9679020047187805 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9054374694824219 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4387456476688385 8"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.365399569272995 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22209544479846954 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 *jet/accuracy_1_of_3‚0.8524700999259949 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1914316862821579 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9780223369598389 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.3056306540966034 particle/accuracy_1_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9855191111564636 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8107360601425171 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9662355780601501 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 particle/accuracy_3_of_7‚1 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4716072380542755 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1317029595375061 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2242063283920288 *jet/accuracy_3_of_5‚0.1108102947473526 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17962400615215302 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22818401455879211 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05038844048976898 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5173728466033936 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9956420660018921 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1681385487318039 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.422746866941452 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16823329031467438 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.47077158093452454 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5366896390914917 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.42096394300460815 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3169659376144409 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *loss/val_total_loss‚46.802512358769214 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3032087981700897 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_6‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1704833060503006 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4143678843975067 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020261500030755997 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9952110648155212 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.902269721031189 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1781938076019287 6 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.330050528049469 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36763638257980347 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1707826554775238 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897217750549316 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 _runtime‚2763.464875632 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4451243579387665 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3315367102622986 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15064555406570435 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48258164525032043 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37972408533096313 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5436355471611023 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.48183420300483704 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.46767616271972656 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2341325581073761 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20362435281276703 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9948107600212097 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3055555820465088 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_6‚1 jet/accuracy_1_of_7‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20779335498809814 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33259862661361694 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01919863000512123 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17614832520484924 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833134651184 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.27961453795433044 *jet/accuracy_1_of_5‚0.9563300013542175 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009247235022485256 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5069444179534912 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5624999403953552 trainer/global_step‚3626 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9929030537605286 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8429490923881531 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3938109874725342 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9874531626701355 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.46785709261894226 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.43333330750465393 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007275132462382317 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13619348406791687 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9665836691856384 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4880951941013336 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.452205091714859 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06955990195274353 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.29655173420906067 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5528802275657654 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.447916716337204 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.516377866268158 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4716072380542755 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.48591697216033936 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23859529197216034 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9908242225646973 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36821702122688293 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397¾K‡A߆§ +jet/accuracy_2_of_2‚0.09123020619153976 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6515151262283325 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.39254388213157654 *jet/accuracy_4_of_5‚0.0098566310480237 particle/accuracy_4_of_5‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2702823281288147 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.46871861815452576 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.36610662937164307 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.19205625355243683 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9646825194358826 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_2_of_2‚1 jet/accuracy_2_of_7‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6738815903663635 jet/accuracy_5_of_5‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.43209877610206604 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3240579068660736 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08389249444007874 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.023002352565526962 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17207400500774384 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_4‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.46992361545562744 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_3‚1 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.32532069087028503 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.41334328055381775 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.483917772769928 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.17241379618644714 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.023972222581505775 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18183264136314392 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.026392610743641853 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1706915944814682 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 particle/accuracy_2_of_5‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03723229095339775 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4579904079437256 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.32575759291648865 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.366666704416275 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.32537201046943665 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23161064088344574 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 particle/accuracy_1_of_1‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4701441526412964 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2399841845035553 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25295308232307434 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5040625333786011 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.058811720460653305 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4896259307861328 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"PuritÈl°þLyy/*FRt3SRqqt0FBt/SRqqt_purity‚0.38930708169937134 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 $ _timestamp‚1.788431078757952e+09 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9893487691879272 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3723905086517334 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5578149557113647 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21281607449054718 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.983271598815918 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.380768746137619 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.44460251927375793 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48325392603874207 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.455828994512558 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4857316017150879 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40435731410980225 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4510582387447357 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.3719954490661621 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0476190485060215 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9809584617614746 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.48576676845550537 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4766666889190674 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4282742440700531 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 epoch‚2 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21729157865047455 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9834064841270447 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4640178680419922 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16003088653087616 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9812644720077515 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1460905373096466 jet/accuracy_4_of_6‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15064555406570435 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5333333611488342 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121212482452393 jet/accuracy_4_of_7‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.41662806272506714 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9863973259925842 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3703703284263611 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7341541051864624 particle/accuracy_2_of_4‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36666664481163025 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5526316165924072 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22574958205223083 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9857913851737976 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3512346148490906 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.022012578323483467 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0778001993894577 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01455671526491642 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 +jet/accuracy_4_of_4‚0.00365606602281332 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36423614621162415 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 jet/accuracy_1_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00382487871684134 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15441717207431793 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21064814925193787 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08525852113962173 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.41542014479637146 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3539038598537445 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12817524373531342 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.03125 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22033430635929108 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9865415692329407 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15486213564872742 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4252593517303467 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 particle/accuracy_1_of_5‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4313177764415741 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8859649300575256 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.025486430153250694 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4246675372123718 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06592261791229248 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.23254819214344025 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.48364463448524475 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9869614839553833 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3672953248023987 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 +jet/accuracy_2_of_4‚0.35793372988700867 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3024691939353943 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952027797698975 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17207247018814087 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4580737054347992 particle/accuracy_3_of_5‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24814818799495697 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 particle/accuracy_5_of_5‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 +jet/accuracy_3_of_6‚0.07692307978868484 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6289891004562378 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3583333194255829 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2358071506023407 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16934022307395935 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9902847409248352 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04813218116760254 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21724137663841248 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24852797389030457 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 _step‚296 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12807594239711761 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20189380645751953 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4841269552707672 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4424987733364105 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12222221493721008 particle/accuracy_2_of_3‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5128205418586731 jet/accuracy_6_of_7‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5361305475234985 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9437602758407593 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.40838780999183655 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5624999403953552 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996392130851746 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3766271471977234 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13432864844799042 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20954936742782593 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4544389843940735 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,particle/sensitivity‚0.29808613657951355 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.98689866065979 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 particle/accuracy_4_of_7‚1 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9679020047187805 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9054374694824219 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4387456476688385 8"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.365399569272995 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22209544479846954 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 *jet/accuracy_1_of_3‚0.8524700999259949 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1914316862821579 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9780223369598389 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.3056306540966034 particle/accuracy_1_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9855191111564636 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8107360601425171 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9662355780601501 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 particle/accuracy_3_of_7‚1 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4716072380542755 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1317029595375061 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2242063283920288 *jet/accuracy_3_of_5‚0.1108102947473526 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17962400615215302 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22818401455879211 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05038844048976898 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5173728466033936 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9956420660018921 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1681385487318039 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.422746866941452 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16823329031467438 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.47077158093452454 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5366896390914917 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.42096394300460815 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3169659376144409 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *loss/val_total_loss‚46.802512358769214 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3032087981700897 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_6‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1704833060503006 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4143678843975067 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020261500030755997 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9952110648155212 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.902269721031189 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1781938076019287 6 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.330050528049469 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36763638257980347 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1707826554775238 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897217750549316 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 _runtime‚2763.464875632 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4451243579387665 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3315367102622986 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15064555406570435 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48258164525032043 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37972408533096313 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5436355471611023 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.48183420300483704 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.46767616271972656 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2341325581073761 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20362435281276703 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9948107600212097 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3055555820465088 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_6‚1 jet/accuracy_1_of_7‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20779335498809814 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33259862661361694 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01919863000512123 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17614832520484924 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833134651184 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.27961453795433044 *jet/accuracy_1_of_5‚0.9563300013542175 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009247235022485256 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5069444179534912 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5624999403953552 trainer/global_step‚3626 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9929030537605286 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8429490923881531 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3938109874725342 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9874531626701355 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.46785709261894226 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.43333330750465393 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007275132462382317 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13619348406791687 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9665836691856384 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4880951941013336 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.452205091714859 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06955990195274353 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.29655173420906067 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5528802275657654 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.447916716337204 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.516377866268158 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4716072380542755 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.48591697216033936 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23859529197216034 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9908242225646973 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36821702122688293 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4511494040489197 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.979583740234375 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.05319149047136307 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5084154009819031 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.44913679361343384 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.43993711471557617 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02139192260801792 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.35625001788139343 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 )jet/accuracy_3_of_4‚0.046409722417593 7validation_average_jet_accuracy‚0.41997653245925903 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009139138273894787 ("Purity/2FRt1SRqqt2FBt/event_purity‚0¨?0²ôžà‚(:“ ò•åÔÑŒŽ gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚162.784' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.556799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×Ý{8á‚(:­ ò•åÔõìõ# proc.memory.rssMB‚ 2870.01953125 network.recv‚ 1.015068e+06 disk.dm-0.in‚0 disk.sda.out‚ 109.61328125 disk.nvme0n1.in‚0& disk.nvme0n1.out‚15610.25341796875 proc.cpu.threads‚17% disk./.usageGB‚1100.6883277893066 disk.rbd4.out‚0 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.51720905303955 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 109.61328125 disk.dm-1.out‚ 109.61328125 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.in‚0( disk.nvme0c0n1.out‚15610.25341796875 disk.rbd4.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0) disk./.usagePercent‚73.86811721696135 disk.rbd1.in‚0 disk.sda2.out‚0$ memory_percent‚94.19467449188232 cpu‚25.368763401369183 network.sent‚ 7.605231e+06 disk.rbd2.out‚ 97.482421875# disk.dm-0.out‚15610.25341796875 disk.sda1.in‚0 disk.rbd1.out‚ 1.533203125 disk.rbd0.in‚0( proc.memory.availableMB‚ 951.14453125ð¦îµ2â- _wandbruntime‚2783  _runtime‚2783Éô–‰ãÀ trainer/global_step‚3649 $ _timestamp‚1.788431098882932e+09 _runtime‚2783.587800913 _step‚297 & lr-AdamW/pg1‚‹J÷R=0.0001996485844936942 & lr-AdamW/pg2‚0.0001996485844936942§ˆ¢4ËäÅ & lr-AdamW/pg1‚0.0001996485844936942 & lr-AdamW/pg2‚0.0001996485844936942 trainer/global_step‚3649 $ _timestamp‚1.788431098882932e+09 _runtime‚2783.587800913 _step‚297© g„¯2å- _wandbruntime‚2783  _runtime‚2783Àâ¢Ý'æ×O JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17580880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232204361417216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213079855616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3061429181952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚3649 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215727808906240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215727808906240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68982099 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215727134540800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122210469 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚215727134540800 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69002137 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207086280475648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚2783.588673598 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚212666379724288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3212903648768 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚212666379724288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53208332 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31048156 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53208330 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17580433 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48629038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210135781661026 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207086297515008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3061429181952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235417441272832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3049501185378 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401< Lâ'çÜO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3049501185378 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17580880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232204361417216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213079855616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3061429181952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚3649 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215727808906240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215727808906240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68982099 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215727134540800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122210469 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚215727134540800 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69002137 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207086280475648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚2783.588673598 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚212666379724288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3212903648768 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚212666379724288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53208332 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31048156 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53208330 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17580433 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48629038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210135781661026 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207086297515008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3061429181952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235417441272832ª¹ è®2è- _wandbruntime‚2784  _runtime‚2784Êe(yç&éáM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215784307007488 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122242965 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122222899 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3062248623104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48641639 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210191502226550 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3062248623104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚3649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122222899 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3050972058306 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235479735370752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69020617 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884310995632217e+09 _runtime‚2784.268608864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69000553 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3213747155968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53222346 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69020617 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53222346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53222348 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3213923349504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3061592247296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17585313 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3050317620342 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122242965 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53222348 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48641180 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3061592247296 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212722714760192 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215784981386240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69000553 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207141184606208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212722714760192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0œ÷?\¶êæM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212722714760192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215784307007488 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122242965 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122222899 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3062248623104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48641639 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210191502226550 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3062248623104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚3649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122222899 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3050972058306 KEDeviceStatsMonitor.on_train_batch_end/r)6ƒJ6eserved_bytes.large_pool.freed‚0 _step‚299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235479735370752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69020617 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884310995632217e+09 _runtime‚2784.268608864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69000553 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3213747155968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53222346 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69020617 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53222346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53222348 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3213923349504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3061592247296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17585313 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3050317620342 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122242965 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53222348 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48641180 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3061592247296 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212722714760192 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215784981386240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69000553 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207141184606208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215484013663744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3101758625280 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17804758 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212922495005550 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31456558 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3255886755328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123837321 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69928189 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69928189 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53909132 _runtime‚2818.751054703 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53909130 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69908151 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209831487003648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209831504043008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235286898300416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215484031666688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215484013663744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚218586446654464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31456556 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235286874360320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3090336524786 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚218585772289024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3102414987776 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218585772289024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49260887 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0dmð( ÷ÛO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218585772289024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49260887 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49261316 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53909130 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3255721034240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚302 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215484013663744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3101758625280 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17804758 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212922495005550 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31456558 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3255886755328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123837321 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69928189 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69928189 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53909132 _runtime‚2818.751054703 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53909130 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69908151 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209831487003648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.sma‡Ôʹll_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209831504043008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235286898300416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215484031666688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215484013663744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚218586446654464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31456556 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235286874360320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3090336524786 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚218585772289024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3102414987776 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3103234428928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17808846 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235348549038080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3091807397714 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53923148 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53923146 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69946669 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49273570 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69946669 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚235348525097984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3103234428928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3256584464384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884311347044983e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚2819.409678849 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212977544093958 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123869817 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123869817 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238605109562368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69926605 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215540384705536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3256745977856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218643619134464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31464726 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31464724 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53923148 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3102578053120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123849751 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844Ï@±Êì&úæM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚2819.409678849 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212977544093958 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123869817 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123869817 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238605109562368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69926605 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215540384705536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3256745977856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218643619134464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31464726 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31464724 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53923148 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3102578053120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123849751 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3103234428928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17808846 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235348549038080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3091807397714 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53923148 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53923146 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69946669 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49273570 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69946669 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚235348525097984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3103234428928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3256584464384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884311347044983e+09¯|2û- _wandbruntime‚2819  _runtime‚2819'd¹UüÏ trainer/global_step‚3699 0loss/FRt2/assignment_loss‚2.9620186674351596 _step‚304 /loss/FRt4/assignment_loss‚3.104749092241613 2loss/SRqqt3/assignment_loss‚1.6609717368015222 0loss/SRqqt2/detection_loss‚2.207898093236459 &loss/total_loss‚46.165230188103294 /loss/FRt3/detection_loss‚2.9510418157423697 _runtime‚2819.410729389 1loss/SRqqt4/assignment_loss‚1.395976367518948 % _timestamp‚1.7884311347061045e+09 0loss/FRt1/assignment_loss‚1.6622485573901686 1loss/FBt1/assignment_loss‚0.25647379270355436 0loss/SRqqt4/detection_loss‚1.775119935644084 /loss/FRt3/assignment_loss‚3.948641869329637 /loss/FRt2/detection_loss‚2.2308900350210616 .loss/FRt4/detection_loss‚2.434242050822188 1loss/SRqqt1/detection_loss‚1.1461633248648173 .loss/FBt2/detection_loss‚2.797686830163002 .loss/FRt1/detection_loss‚2.096817040859267 0loss/SRqqt3/detection_loss‚2.852114535868168 /loss/FBt3/detection_loss‚3.3058958500623703 epoch‚3 2loss/SRqqt1/assignment_loss‚0.8526558398046125 2loss/SRqqt2/assignment_loss‚1.5741907736739598 0loss/FBt2/assignment_loss‚0.3872614235926335 /loss/FBt3/assignment_loss‚0.688613070640713 0loss/FBt4/assignment_loss‚0.5827178852632642 .loss/FBt1/detection_loss‚0.743941237647374 .loss/FBt4/detection_loss‚2.546900361776352LfvNZýÔ /loss/FRt3/assignment_loss‚3.948641869329637 /loss/FRt2/detection_loss‚2.2308900350210616 .loss/FRt4/detection_loss‚2.434242050822188 1loss/SRqqt1/detection_loss‚1.1461633248648173 .loss/FBt2/detection_loss‚2.797686830163002 .loss/FRt1/detection_loss‚2.096817040859267 0loss/SRqqt3/detection_loss‚2.852114535868168 /loss/FBt3/detection_loss‚3.3058958500623703 epoch‚3 2loss/SRqqt1/assignment_loss‚0.8526558398046125 2loss/SRqqt2/assignment_loss‚1.5741907736739598 0loss/FBt2/assignment_loss‚0.3872614235926335 /loss/FBt3/assignment_loss‚0.688613070640713 0loss/FBt4/assignment_loss‚0.5827178852632642 .loss/FBt1/detection_loss‚0.743941237647374 .loss/FBt4/detection_loss‚2.546900361776352 trainer/global_step‚3699 0loss/FRt2/assignment_loss‚2.9620186674351596 _step‚304 /loss/FRt4/assignment_loss‚3.104749092241613 2loss/SRqqt3/assignment_loss‚1.6609717368015222 0loss/SRqqt2/detection_loss‚2.207898093236459 &loss/total_loss‚46.165230188103294 /loss/FRt3/detection_loss‚2.9510418157423697 _runtime‚2819.410729389 1loss/SRqqt4/assignment_loss‚1.395976367518948 % _timestamp‚1.7884311347061045e+09 0loss/FRt1/assignment_loss‚1.6622485573901686 1loss/FBt1/assignment_loss‚0.25647379270355436 0loss/SRqqt4/detection_loss‚1.775119935644084°suÎ5žþ‚(:“ Ÿ–åÔ«Ôû gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.691' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.138199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é0(ÿ‚(: Ÿ–åÔ­Ê­% disk./.usageGB‚1100.6884498596191 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.dm-1.out‚111.5625 disk.rbd2.out‚97.60791015625 cpu‚25.363645856697477 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 1.533203125 proc.cpu.threads‚17 network.recv‚ 1.032767e+06 disk.sda3.in‚0! disk.dm-0.out‚15908.357421875 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda.out‚111.5625 disk.nvme0n1.in‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚15908.357421875 disk.sda2.out‚0 network.sent‚ 7.843999e+06 disk.nvme0c0n1.in‚0 disk.rbd2.in‚0) proc.memory.percent‚17.51720905303955 disk.sda3.out‚111.5625 disk.rbd3.in‚0 disk.dm-0.in‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚15908.357421875$ memory_percent‚94.19353008270264( proc.memory.availableMB‚ 951.33203125# proc.memory.rssMB‚ 2870.01953125) disk./.usagePercent‚73.86812540920245 disk.sda.in‚0 disk.rbd1.in‚0¢bP “€‚(:ˆ ®–åÔÅ”’ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.947' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.7894 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^‚§ç2‚(:§ ®–åÔƒ‘ disk.dm-0.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 1.533203125 disk.sda3.in‚0 disk.sda3.out‚ 112.25390625 disk.rbd3.out‚0 network.sent‚ 7.95181e+06 disk.rbd2.out‚97.67041015625 cpu‚25.362005518690378 disk.sda2.out‚0 disk.rbd4.in‚0% disk./.usageGB‚1100.6881523132324 disk.sda.out‚ 112.25390625 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚16007.458984375 disk.sda2.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2870.01953125) disk./.usagePercent‚73.86810544061477 disk.dm-1.in‚0& disk.nvme0c0n1.out‚16007.458984375 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0$ memory_percent‚94.19312477111816) proc.memory.percent‚17.51720905303955 network.recv‚ 1.038453e+06 disk.dm-1.out‚ 112.25390625 disk.rbd1.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17! disk.dm-0.out‚16007.458984375 disk.sda1.out‚0 disk.nvme0c0n1.in‚0 disk.drbd1777.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 951.3984375éËÁì‚‚(:’ ½–åÔç芎 gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.05' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.410000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡•|ñ)ƒ‚(:ž ½–åÔ«‹‘‘) proc.memory.percent‚17.51720905303955 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda.in‚0% proc.memory.availableMB‚ 951.15625$ disk./.usageGB‚1100.688133239746 disk.rbd1.out‚ 1.533203125 disk.sda3.out‚ 112.8828125 disk.rbd3.in‚0! disk.dm-0.out‚16007.904296875 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚16007.904296875 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.rbd4.out‚0# memory_percent‚94.1946029663086# proc.memory.rssMB‚ 2870.01953125& disk.nvme0c0n1.out‚16007.904296875 disk.rbd2.in‚0 disk.rbd2.out‚99.19775390625 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 112.8828125 disk.dm-0.in‚0 disk.sda.out‚ 112.8828125 cpu‚25.36027320153376 network.recv‚ 1.043023e+06 disk.nvme0c0n1.in‚0 network.sent‚ 7.95825e+06( disk./.usagePercent‚73.8681041605771 disk.dm-1.in‚0 disk.sda3.in‚0Ó—2À2„- _wandbruntime‚2854  _runtime‚2854jæ”É…Ã trainer/global_step‚3749 % _timestamp‚1.7884311697797585e+09 _runtime‚2854.484485655 _step‚305 ' lr-AdamW/pg1‚0.00019749410208533868 ' lr-AdamW/pg2‚0.00019749410208533868»ÃG£Î†È ' lr-AdamW/pg1‚0.00019749410208533868 ' lr-AdamW/pg2‚0.00019749410208533868 trainer/global_step‚3749 % _timestamp‚1.7884311697797585e+09 _runtime‚2854.484485655 _step‚305±§à«Š2‡- _wandbruntime‚2854  _runtime‚2854ÖPô— ˆÕO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221444410037248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 2854.49276042 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70834203 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54609932 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54609930 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 [IDeviceStatsMonitor.on_train_batch‰¾|ÞD_start/active_bytes.small_pool.allocated‚ 3143400793600 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54609932 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212576710571008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31864956 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚238369435183616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3298663606272 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3142744431104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚218301683609088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31864958 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚218301665606144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3298491593728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.788431169780457e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3142744431104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3131826301950 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚221445084402688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3143400793600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18030655 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18030219 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215707865395842 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125444133 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚218301665606144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚221445084402688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221444410037248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚218301683609088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125464173 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241668098789888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49895613 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70834203 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241667902837248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚306 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70854241 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70854241 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944FÃݲà'‰ÚO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚218301683609088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31864958 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚218301665606144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3298491593728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.788431169780457e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3142744431104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3131826301950 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚221445084402688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3143400793600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18030655 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18030219 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215707865395842 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125444133 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚218301665606144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚221445084402688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221444410037248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚218301683609088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125464173 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241668098789888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49895613 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70834203 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241667902837248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚306 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70854241 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70854241 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221444410037248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 2854.49276042 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70834203 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54609932 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54609930 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3143400793600 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54609932 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212576710571008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31864956 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚238369435183616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3298663606272 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3142744431104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128²õ‡æ2Š- _wandbruntime‚2855  _runtime‚2855ç°Wæ&‹àM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3144220234752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚2855.165826176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3144220234752 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚221502256882688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚215763585961366 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚218358036647936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3131988299158 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31873126 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚218358036647936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3143563858944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18035083 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚3749 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221502256882688 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚238431085921280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54623948 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚307 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54623946 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3143563858944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3299512457728 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49907766 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125496669 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 $ _timestamp‚1.788431170460424e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125476603 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70852657 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70872721 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241730598379008 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31873124 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2ŽìmÊ¡ŒåM PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49907766 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125496669 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 $ _timestamp‚1.788431170460424e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125476603 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70852657 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70872721 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241730598379008 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31873124 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3144220234752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚2855.165826176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3144220234752 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚221502256882688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚215763585961366 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚218358036647936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3131988299158 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31873126 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚218358036647936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3143563858944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18035083 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚3749 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221502256882688 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚238431085921280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54623948 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚307 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54623946 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3143563858944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3299512457728 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0³Þíü2- _wandbruntime‚2855  _runtime‚2855 Á=YŽÓ .loss/FBt3/detection_loss‚2.790151302631085 /loss/FRt2/assignment_loss‚2.878103642891615 %loss/total_loss‚46.95467261043751 trainer/global_step‚3749 _step‚308 1loss/SRqqt1/detection_loss‚1.2392473991960287 2loss/SRqqt2/assignment_loss‚2.0175038785846144 /loss/FRt2/detection_loss‚2.7089656048860307 2loss/SRqqt3/assignment_loss‚1.8040968161076307 2loss/SRqqt4/assignment_loss‚1.5803468400502907 /loss/FRt1/detection_loss‚1.7274637913756203 .loss/FBt2/detection_loss‚4.196511417627335 _runtime‚2855.166815982 0loss/FRt1/assignment_loss‚1.6435195040284543 .loss/FRt3/detection_loss‚3.338577491896493 % _timestamp‚1.7884311704621663e+09 0loss/FRt4/assignment_loss‚3.6962092116821643 /loss/FBt1/detection_loss‚0.5863633578706972 .loss/FBt4/detection_loss‚2.579664956439625 epoch‚3 /loss/FRt3/assignment_loss‚3.908028210912432 1loss/FBt1/assignment_loss‚0.17584497444525504 0loss/FBt3/assignment_loss‚0.2933773798151658 .loss/FRt4/detection_loss‚2.180692264506983 1loss/SRqqt3/detection_loss‚2.4478405211120844 0loss/SRqqt4/detection_loss‚1.783159421939476 2loss/SRqqt1/assignment_loss‚1.2281434267014264 0loss/FBt2/assignment_loss‚0.1606214316561818 1loss/FBt4/assignment_loss‚0.20714416448026896 1loss/SRqqt2/detection_loss‚1.7830955996005624Ùå^Ø 2loss/SRqqt3/assignment_loss‚1.8040968161076307 2loss/SRqqt4/assignment_loss‚1.5803468400502907 /loss/FRt1/detection_loss‚1.7274637913756203 .loss/FBt2/detection_loss‚4.196511417627335 _runtime‚2855.166815982 0loss/FRt1/assignment_loss‚1.6435195040284543 .loss/FRt3/detection_loss‚3.338577491896493 % _timestamp‚1.7884311704621663e+09 0loss/FRt4/assignment_loss‚3.6962092116821643 /loss/FBt1/detection_loss‚0.5863633578706972 .loss/FBt4/detection_loss‚2.579664956439625 epoch‚3 /loss/FRt3/assignment_loss‚3.908028210912432 1loss/FBt1/assignment_loss‚0.17584497444525504 0loss/FBt3/assignment_loss‚0.2933773798151658 .loss/FRt4/detection_loss‚2.180692264506983 1loss/SRqqt3/detection_loss‚2.4478405211120844 0loss/SRqqt4/detection_loss‚1.783159421939476 2loss/SRqqt1/assignment_loss‚1.2281434267014264 0loss/FBt2/assignment_loss‚0.1606214316561818 1loss/FBt4/assignment_loss‚0.20714416448026896 1loss/SRqqt2/detection_loss‚1.7830955996005624 .loss/FBt3/detection_loss‚2.790151302631085 /loss/FRt2/assignment_loss‚2.878103642891615 %loss/total_loss‚46.95467261043751 trainer/global_step‚3749 _step‚308 1loss/SRqqt1/detection_loss‚1.2392473991960287 2loss/SRqqt2/assignment_loss‚2.0175038785846144 /loss/FRt2/detection_loss‚2.7089656048860307´éôÆõ‚(:’ Ì–åÔïøŽ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.54' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.907999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ggo(‘‚(: Ì–åÔ¢šó disk.sda2.in‚0 disk.sda.out‚ 114.4296875& disk.nvme0n1.out‚16112.96533203125 disk.rbd1.in‚0( proc.memory.availableMB‚ 1026.8359375 disk.rbd1.out‚ 1.533203125 network.recv‚ 1.04695e+06 disk.dm-0.in‚ 0.17578125 disk.nvme0n1.in‚ 0.17578125$ memory_percent‚93.73269081115723 disk.rbd2.out‚99.26025390625 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.86814844988052 disk.dm-1.in‚0( disk.nvme0c0n1.out‚16112.96533203125 cpu‚25.358667342137615 disk.sda3.out‚ 114.4296875# disk.dm-0.out‚16112.96533203125 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 114.4296875* proc.memory.percent‚17.055296897888184 disk.rbd0.out‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 0.17578125 disk.rbd2.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2794.33984375 network.sent‚ 7.963664e+06$ disk./.usageGB‚1100.688793182373$;£“’‚(:ˆ Û–åÔÑåî gpu.0.gpu‚40.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚117.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„LRr“‚(:Ž Û–åÔ·³™ disk.sda3.out‚ 117.046875 disk.rbd3.in‚0 disk.sda1.in‚0* proc.memory.percent‚17.055296897888184 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.87313803672286 disk.dm-1.out‚ 117.046875 disk.nvme0c0n1.in‚0.578125 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.drbd1777.out‚0" disk.dm-0.out‚16291.0947265625 disk.sda.out‚ 117.046875 disk.dm-1.in‚0' disk.nvme0c0n1.out‚16291.0947265625 disk.sda2.in‚0& proc.memory.availableMB‚ 1027.09375 cpu‚25.35708693153698 disk.rbd2.in‚0 disk.dm-0.in‚0.578125 disk.rbd1.out‚ 1.533203125$ memory_percent‚93.73111724853516# proc.memory.rssMB‚ 2794.33984375 network.sent‚ 8.071585e+06 network.recv‚ 1.052784e+06 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda.in‚0% disk.nvme0n1.out‚16291.0947265625# disk./.usageGB‚1100.76314163208 disk.rbd2.out‚99.27587890625 disk.nvme0n1.in‚0.578125 Á2”- _wandbruntime‚2889  _runtime‚2889ï…óµÉ•Ã ' lr-AdamW/pg2‚0.00019639043238527156 trainer/global_step‚3799 % _timestamp‚1.7884312051073263e+09 _runtime‚2889.812040266 _step‚309 ' lr-AdamW/pg1‚0.00019639043238527156ÊLê“Î–È % _timestamp‚1.7884312051073263e+09 _runtime‚2889.812040266 _step‚309 ' lr-AdamW/pg1‚0.00019639043238527156 ' lr-AdamW/pg2‚0.00019639043238527156 trainer/global_step‚3799µÔ¶‹Ç2—- _wandbruntime‚2889  _runtime‚2889ÂÓÁNÝ'˜×O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71780293 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71780293 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3183730236928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221119335551488 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221119335551488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18257052 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3172007203602 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244793543762432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3341395488768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55310732 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55310732 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3341571695616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71760255 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218494578740366 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32273358 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224303047785472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55310730 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3183730236928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50530410 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚2889.812979807 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244793343615488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884312051080124e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71780293 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3183730236928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221119335551488 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221119335551488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18257052 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3172007203602 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244793543762432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3341395488768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55310732 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55310732 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3341571695616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71760255 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218494578740366 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32273358 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224303047785472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55310730 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3183730236928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50530410 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚2889.812979807 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244793343615488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884312051080124e+09 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55324748 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71798773 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32281526 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884312057909057e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 JBDeviceStatsMon KÚç!itor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚3799 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221175670587392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3185206040576 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241513622804480 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50542646 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127103455 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218550299306098 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127123521 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244856036732416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3342413927936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚224360894630912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18261555 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221175670587392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18261122 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215376821229568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3172823638566 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3184549664768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55324748 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221175688590336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244855838695936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241513598864384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71778709 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3184549664768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3342239831552 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚311 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71778709 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32281524 B&ì&œæM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3184549664768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55324748 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221175688590336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244855838695936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241513598864384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71778709 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3184549664768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3342239831552 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚311 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71778709 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32281524 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55324748 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71798773 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32281526 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884312057909057e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚3799 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221175670587392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3185206040576 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241513622804480 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50542646 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127103455 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218550299306098 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127123521 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244856036732416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3342413927936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚224360894630912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18261555 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221175670587392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18261122 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215376821229568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3172823638566·"d ¤2- _wandbruntime‚2890  _runtime‚2890å¹ÐŠYžÓ epoch‚3 _runtime‚2890.497335389 2loss/SRqqt1/assignment_loss‚1.1102600830296676 1loss/SRqqt1/detection_loss‚1.2017243952800831 0loss/FRt1/assignment_loss‚1.7606526280824955 2loss/SRqqt4/assignment_loss‚1.5853658449547043 /loss/FRt4/detection_loss‚2.1579084770245984 1loss/SRqqt4/detection_loss‚1.6695794899915826 /loss/FRt1/detection_loss‚1.9408763140583267 -loss/FRt2/detection_loss‚2.57694682140242 1loss/SRqqt2/detection_loss‚1.9802057882770896 _step‚312 0loss/FRt3/assignment_loss‚3.6851385712623594 0loss/FRt4/assignment_loss‚3.1636018926447087 2loss/SRqqt2/assignment_loss‚1.6856888228406508 0loss/FBt4/assignment_loss‚0.6739953370256857 /loss/FBt1/detection_loss‚0.6885319621715097 0loss/FBt2/assignment_loss‚0.3873306911970888 0loss/SRqqt3/detection_loss‚2.733930232003331 % _timestamp‚1.7884312057925985e+09 2loss/SRqqt3/assignment_loss‚2.1473208204843104 0loss/FBt3/assignment_loss‚0.3798672169446642 /loss/FRt3/detection_loss‚3.0324393848578137 $loss/total_loss‚46.2129673146474 trainer/global_step‚3799 0loss/FRt2/assignment_loss‚3.1730512218041853 1loss/FBt1/assignment_loss‚0.22286647754723843 /loss/FBt2/detection_loss‚2.9747045040130615 -loss/FBt3/detection_loss‚2.90858256816864 .loss/FBt4/detection_loss‚2.372397769581188:{à^ŸØ 2loss/SRqqt1/assignment_loss‚1.1102600830296676 1loss/SRqqt1/detection_loss‚1.2017243952800831 0loss/FRt1/assignment_loss‚1.7606526280824955 2loss/SRqqt4/assignment_loss‚1.5853658449547043 /loss/FRt4/detection_loss‚2.1579084770245984 1loss/SRqqt4/detection_loss‚1.6695794899915826 /loss/FRt1/detection_loss‚1.9408763140583267 -loss/FRt2/detection_loss‚2.57694682140242 1loss/SRqqt2/detection_loss‚1.9802057882770896 _step‚312 0loss/FRt3/assignment_loss‚3.6851385712623594 0loss/FRt4/assignment_loss‚3.1636018926447087 2loss/SRqqt2/assignment_loss‚1.6856888228406508 0loss/FBt4/assignment_loss‚0.6739953370256857 /loss/FBt1/detection_loss‚0.6885319621715097 0loss/FBt2/assignment_loss‚0.3873306911970888 0loss/SRqqt3/detection_loss‚2.733930232003331 % _timestamp‚1.7884312057925985e+09 2loss/SRqqt3/assignment_loss‚2.1473208204843104 0loss/FBt3/assignment_loss‚0.3798672169446642 /loss/FRt3/detection_loss‚3.0324393848578137 $loss/total_loss‚46.2129673146474 trainer/global_step‚3799 0loss/FRt2/assignment_loss‚3.1730512218041853 1loss/FBt1/assignment_loss‚0.22286647754723843 /loss/FBt2/detection_loss‚2.9747045040130615 -loss/FBt3/detection_loss‚2.90858256816864 .loss/FBt4/detection_loss‚2.372397769581188 epoch‚3 _runtime‚2890.497335389¸¾ô£’ ‚(:‡ ê–åÔŠæß gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚90.353' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0706 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“Ðz!¡‚(:– ê–åÔü«ú disk.sda1.in‚0 disk.dm-1.out‚ 117.61328125' disk.nvme0c0n1.out‚16390.1220703125 disk.sda.in‚0$ memory_percent‚93.73140335083008' proc.memory.availableMB‚ 1027.046875* proc.memory.percent‚17.055296897888184 disk.rbd0.out‚0% disk./.usageGB‚1100.7631225585938 disk.sda3.in‚0 disk.sda3.out‚ 117.61328125 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 117.61328125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚0.578125# proc.memory.rssMB‚ 2794.33984375 disk.nvme0c0n1.in‚0.578125 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚0.578125" disk.dm-0.out‚16390.1220703125 disk.sda1.out‚0 network.sent‚ 8.076924e+06 network.recv‚ 1.056444e+06% disk.nvme0n1.out‚16390.1220703125 disk.rbd1.in‚0 disk.rbd2.out‚ 99.3701171875 disk.dm-1.in‚0 cpu‚25.355514502966127( disk./.usagePercent‚73.8731367566852 disk.rbd1.out‚ 1.533203125 disk.rbd0.in‚0 disk.drbd1777.out‚0—y%o“¢‚(:ˆ ù–åÔ®¶ó gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³Ñ#£‚(:˜ ù–åÔ†·) disk./.usagePercent‚73.87509700637487 disk.sda2.in‚0 disk.drbd1777.in‚0* proc.memory.percent‚17.055296897888184% disk./.usageGB‚1100.7923316955566 disk.dm-0.in‚ 6.69140625$ memory_percent‚93.73054504394531 disk.sda2.out‚0 disk.sda.out‚ 118.2265625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 6.69140625' disk.nvme0c0n1.out‚16400.7978515625% proc.memory.availableMB‚ 1027.1875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 118.2265625 disk.rbd3.in‚0 disk.sda1.in‚0 network.recv‚ 1.063259e+06 disk.rbd2.out‚ 99.3857421875% disk.nvme0n1.out‚16400.7978515625# proc.memory.rssMB‚ 2794.33984375 network.sent‚ 8.185991e+06 disk.rbd3.out‚0 disk.dm-1.out‚ 118.2265625 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 1.533203125 cpu‚25.353944972308337 disk.rbd0.in‚0" disk.dm-0.out‚16400.7978515625 disk.sda.in‚0 disk.nvme0c0n1.in‚ 6.69140625LXÏvž¤‚(:“ ˆ—åÔÿÃÀ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.329' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.665800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û´æ"¥‚(:— ˆ—åÔÎÛÑ disk.rbd2.in‚0 disk.dm-1.out‚ 118.734375 disk.rbd1.in‚0' disk.nvme0c0n1.out‚16519.3447265625) proc.memory.availableMB‚ 1026.93359375 network.sent‚ 8.19142e+06 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚93.73209476470947 disk.nvme0c0n1.in‚ 7.22265625 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0) disk./.usagePercent‚73.87451894136255% disk./.usageGB‚1100.7837181091309 disk.rbd2.out‚ 99.4482421875 disk.rbd1.out‚ 1.533203125 disk.sda2.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2794.33984375" disk.dm-0.out‚16519.3447265625 disk.sda.out‚ 118.734375% disk.nvme0n1.out‚16519.3447265625 network.recv‚ 1.066988e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 118.734375 disk.dm-0.in‚ 7.22265625 disk.nvme0n1.in‚ 7.22265625 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.055296897888184 cpu‚25.35239658462498 proc.cpu.threads‚17·dÑÍ2¦- _wandbruntime‚2925  _runtime‚2925ƒ´ßɧà ' lr-AdamW/pg1‚0.00019526944914007694 ' lr-AdamW/pg2‚0.00019526944914007694 trainer/global_step‚3849 % _timestamp‚1.7884312407308295e+09 _runtime‚2925.435552758 _step‚313aLOúÎ¨È % _timestamp‚1.7884312407308295e+09 _runtime‚2925.435552758 _step‚313 ' lr-AdamW/pg1‚0.00019526944914007694 ' lr-AdamW/pg2‚0.00019526944914007694 trainer/global_step‚3849¹RÈ\v2©- _wandbruntime‚2925  _runtime‚2925.ݱ°ª×O O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227162359899136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223936969490944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56011530 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128697837 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3225372405248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72706345 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3225372405248 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223936987493888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32681756 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223936987493888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚3849 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18484163 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56011532 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223936969490944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247918671059456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51165921 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3384347576320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3224716042752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221279949130658 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884312407315435e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128717877 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72686307 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3213496980766 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72706345 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227161685533696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18483729 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218067123627008©Ë‹%â'«ÜO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223936969490944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247918671059456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51165921 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3384347576320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3224716042752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221279949130658 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884312407315435e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128717877 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72686307 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3213496980766 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72706345 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227161685533696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18483729 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218067123627008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227162359899136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223936969490944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56011530 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128697837 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3225372405248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72706345 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3225372405248 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223936987493888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32681756 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223936987493888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚3849 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18484163 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56011532 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944º¾MWŸ2¬- _wandbruntime‚2926  _runtime‚2926=lHç&­áM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223993340532736 % _timestamp‚1.7884312414028006e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72704761 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56025546 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56025546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3214313415938 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72724825 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223993322529792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247981367677440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32689926 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18488220 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56025548 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56025548 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51178144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221336341173506 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72724825 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244596135747584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221335669696182 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51178592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3213658977974 trainer/global_step‚3849 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32689924 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 _step‚315 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72704761 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223993322529792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128750373 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ê/?Iµ!®æM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221335669696182 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51178592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3213658977974 trainer/global_step‚3849 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32689924 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 _step‚315 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72704761 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223993322529792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128750373 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223993340532736 % _timestamp‚1.7884312414028006e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72704761 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56025546 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56025546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3214313415938 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72724825 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223993322529792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247981367677440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32689926 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18488220 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56025548 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56025548 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51178144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221336341173506 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72724825 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244596135747584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0»Ê:ÎÕ2¯- _wandbruntime‚2926  _runtime‚2926îF[°Õ /loss/FRt3/assignment_loss‚4.084580340385437 /loss/FRt3/detection_loss‚3.0141423320770264 0loss/FBt3/assignment_loss‚0.4238528609275818 /loss/FBt3/detection_loss‚3.4490413516759872 &loss/total_loss‚48.522000101712884 % _timestamp‚1.7884312414045403e+09 _runtime‚2926.109480672 2loss/SRqqt4/assignment_loss‚1.6296146564344143 trainer/global_step‚3849 1loss/SRqqt2/assignment_loss‚1.682081542512838 0loss/SRqqt4/detection_loss‚2.209596614888374 /loss/FBt1/detection_loss‚0.5658975048634877 epoch‚3 0loss/FBt2/assignment_loss‚0.3182499729737174 1loss/SRqqt2/detection_loss‚2.2044352876379136 1loss/SRqqt3/detection_loss‚2.6000326468098547 .loss/FBt2/detection_loss‚2.843408949673176 /loss/FBt4/detection_loss‚2.9428254876817976 0loss/FRt2/assignment_loss‚2.6156425508292944 /loss/FRt2/detection_loss‚2.7180184109790906 -loss/FRt4/detection_loss‚2.59615686213648 1loss/SRqqt1/detection_loss‚1.0231025623907517 0loss/FRt1/assignment_loss‚1.5586596014452916 1loss/FBt1/assignment_loss‚0.20149017839786823 /loss/FRt1/detection_loss‚1.6753086171240783 0loss/FRt4/assignment_loss‚3.9160036219132914 2loss/SRqqt1/assignment_loss‚1.4256328054788439 1loss/SRqqt3/assignment_loss‚2.340683919287497 1loss/FBt4/assignment_loss‚0.48354142318878857 _step‚316Јü`±Ú /loss/FRt3/assignment_loss‚4.084580340385437 /loss/FRt3/detection_loss‚3.0141423320770264 0loss/FBt3/assignment_loss‚0.4238528609275818 /loss/FBt3/detection_loss‚3.4490413516759872 &loss/total_loss‚48.522000101712884 % _timestamp‚1.7884312414045403e+09 _runtime‚2926.109480672 2loss/SRqqt4/assignment_loss‚1.6296146564344143 trainer/global_step‚3849 1loss/SRqqt2/assignment_loss‚1.682081542512838 0loss/SRqqt4/detection_loss‚2.209596614888374 /loss/FBt1/detection_loss‚0.5658975048634877 epoch‚3 0loss/FBt2/assignment_loss‚0.3182499729737174 1loss/SRqqt2/detection_loss‚2.2044352876379136 1loss/SRqqt3/detection_loss‚2.6000326468098547 .loss/FBt2/detection_loss‚2.843408949673176 /loss/FBt4/detection_loss‚2.9428254876817976 0loss/FRt2/assignment_loss‚2.6156425508292944 /loss/FRt2/detection_loss‚2.7180184109790906 -loss/FRt4/detection_loss‚2.59615686213648 1loss/SRqqt1/detection_loss‚1.0231025623907517 0loss/FRt1/assignment_loss‚1.5586596014452916 1loss/FBt1/assignment_loss‚0.20149017839786823 /loss/FRt1/detection_loss‚1.6753086171240783 0loss/FRt4/assignment_loss‚3.9160036219132914 2loss/SRqqt1/assignment_loss‚1.4256328054788439 1loss/SRqqt3/assignment_loss‚2.340683919287497 1loss/FBt4/assignment_loss‚0.48354142318878857 _step‚316¼Æô¯I‘²‚(:† ——åÔþÞ”Ž gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.87' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µŽÈË-³‚(:¢ ——åÔÈŸ…‘ network.sent‚ 8.197691e+06 disk.dm-1.out‚ 119.41015625 disk.sda.out‚ 119.41015625* proc.memory.percent‚17.055296897888184 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd2.out‚100.9755859375 disk.drbd1777.in‚0 disk.rbd3.out‚0 cpu‚25.35077540198871& disk.nvme0n1.out‚16621.54443359375 disk.rbd0.in‚0$ memory_percent‚93.73424053192139 network.recv‚ 1.071395e+06 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda.in‚0) disk./.usagePercent‚73.87451766132489 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 7.22265625 disk.sda2.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 119.41015625) proc.memory.availableMB‚ 1026.58203125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 7.22265625 disk.rbd1.out‚ 1.533203125( disk.nvme0c0n1.out‚16621.54443359375 disk.rbd0.out‚0 disk.dm-0.in‚ 7.22265625# proc.memory.rssMB‚ 2794.33984375% disk./.usageGB‚1100.7836990356445# disk.dm-0.out‚16621.54443359375©7팴‚(:’ ¦—åÔ½ÿŒŽ gpu.0.gpu‚5.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.378' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.875599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô¢óo&µ‚(:› ¦—åÔ¶ï disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 119.8046875 disk.dm-0.in‚ 7.24609375* proc.memory.percent‚17.055296897888184 disk.dm-1.out‚ 119.8046875 disk.nvme0c0n1.in‚ 7.24609375 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 119.8046875 cpu‚25.349265013967212) disk./.usagePercent‚73.87452150143791% disk.nvme0n1.out‚16621.9287109375 disk.rbd2.in‚0 disk.rbd2.out‚101.0458984375" disk.dm-0.out‚16621.9287109375 proc.cpu.threads‚17 disk.rbd3.out‚0) proc.memory.availableMB‚ 1026.51171875 disk.nvme0n1.in‚ 7.24609375$ memory_percent‚93.73466968536377 disk.sda1.out‚0 disk.rbd1.out‚ 1.533203125 disk.rbd0.out‚0# proc.memory.rssMB‚ 2794.33984375 network.sent‚ 8.305409e+06% disk./.usageGB‚1100.7837562561035 disk.rbd1.in‚0' disk.nvme0c0n1.out‚16621.9287109375 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 network.recv‚ 1.0769e+06 disk.dm-1.in‚0 disk.rbd3.in‚05V{j2¶- _wandbruntime‚2961  _runtime‚2961ßNjÈÉ·Ã ' lr-AdamW/pg2‚0.00019413138596098288 trainer/global_step‚3899 % _timestamp‚1.7884312763470244e+09 _runtime‚2961.051821906 _step‚317 ' lr-AdamW/pg1‚0.00019413138596098288O¦x¨Î¸È trainer/global_step‚3899 % _timestamp‚1.7884312763470244e+09 _runtime‚2961.051821906 _step‚317 ' lr-AdamW/pg1‚0.00019413138596098288 ' lr-AdamW/pg2‚0.00019413138596098288½ÐúöÑ2¹- _wandbruntime‚2961  _runtime‚2961x¹é Ý'º×O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73632397 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚247617021893120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3254332320174 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚230020323281920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73632397 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220812313115648 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3265701848576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33090158 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247617045833216 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3266358211072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230020323281920 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚226754621433344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251044411097088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56712330 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18711371 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3427365263872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226754639436288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56712332 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3266358211072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251044219338752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224065990998066 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56712332 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226754621433344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73612359 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18710909 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3253677882418 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51801529 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33090156 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3266358211072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230020323281920 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚226754621433344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251044411097088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56712330 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18711371 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3427365263872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226754639436288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56712332 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3266358211072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251044219338752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224065990998066 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56712332 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226754621433344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73612359 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18710909 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3253677882418 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51801529 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33090156 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3254494317382 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226810974472192 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130357159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18715409 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230078170127360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3267177652224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56726346 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33098326 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130377225 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251106717665280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130357159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3428045034496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freb”‹èed‚230077495748608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73630813 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3266521276416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73630813 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220867234285568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56726348 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚2961.723213937 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130377225 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚224122383040914 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚224121711563590 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56726348 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3267177652224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247678672630784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56726346 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51813733 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51814195 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚3899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3254494317382 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226810974472192 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130357159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18715409 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230078170127360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3267177652224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56726346 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33098326 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130377225 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251106717665280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130357159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3428045034496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230077495748608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73630813 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3266521276416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73630813 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220867234285568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56726348 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚2961.723213937 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130377225 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚224122383040914 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚224121711563590 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56726348 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3267177652224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247678672630784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56726346 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51813733 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51814195 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚3899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33498558 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226852032865474 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232878961030144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52436517 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18938388 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250699558776320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3306687654400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74558449 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57413130 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚223557536683008 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57413130 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3307344016896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232878961030144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3307344016896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254169834364416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226852704342590 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229572273375744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229572273375744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3470251648000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884313125464678e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33498556 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232879635395584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232879635395584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚322 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52436946 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3470094315520 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57413132 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74538411 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250699582716416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74558449 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229572291378688 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229572291378688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3306687654400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3295167659582 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3294513221826 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _runtime‚2997.251465876 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496²ææ´ËÜO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57413130 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3307344016896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_byteµ ­.$s.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232878961030144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3307344016896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254169834364416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226852704342590 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229572273375744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229572273375744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3470251648000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884313125464678e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33498556 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232879635395584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232879635395584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚322 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52436946 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3470094315520 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57413132 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74538411 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250699582716416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74558449 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229572291378688 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229572291378688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3306687654400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3295167659582 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3294513221826 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _runtime‚2997.251465876 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18937961 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚254169653091840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131971581 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33498558 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226852032865474 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232878961030144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52436517 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18938388 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250699558776320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3306687654400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74558449 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57413130 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚223557536683008 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0¼ÑMÕ2Ì- _wandbruntime‚2997  _runtime‚2997LóÞå&ÍßM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3307507082240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52449232 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132004077 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57427148 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74576929 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57427146 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250761209513984 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232936807875584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18942953 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229628644417536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3308163458048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74556865 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚323 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3471116173824 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254232349627904 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3295984094754 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 # _timestamp‚1.78843131320945e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33506724 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223612423774208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3949 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131984011 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74556865 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3295329656790 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3307507082240 _runtime‚2997.914976256 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223612440813568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132004077 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0cà*Ëê&ÎäM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226908424908322 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3308163458048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131984011 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57427146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132004077 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3307507082240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52449232 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132004077 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57427148 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74576929 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57427146 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250761209513984 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232936807875584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18942953 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229628644417536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3308163458048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74556865 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚323 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3471116173824 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254232349627904 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3295984094754 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 # _timestamp‚1.78843131320945e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33506724 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223612423774208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3949 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131984011 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74556865 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3295329656790 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3307507082240 _runtime‚2997.914976256 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223612440813568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 P݇êׂ(:‘ ñ—åԜΙ disk.sda3.in‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚98.65625% disk.nvme0n1.out‚17019.6201171875 disk.rbd3.out‚0 network.recv‚ 1.102717e+06 disk.sda3.out‚ 122.30859375 disk.rbd3.in‚0 disk.nvme0c0n1.in‚98.65625 cpu‚25.341850254945196 disk.rbd0.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 1.533203125' disk.nvme0c0n1.out‚17019.6201171875( proc.memory.availableMB‚ 551.86328125 proc.cpu.threads‚17) disk./.usagePercent‚73.87451151714407 disk.dm-0.in‚98.65625 disk.sda1.in‚0$ memory_percent‚96.63169384002686 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0! disk.rbd2.out‚102.83544921875 disk.sda2.out‚0 proc.memory.rssMB‚2794.5( proc.memory.percent‚17.0562744140625 network.sent‚ 8.540317e+06% disk./.usageGB‚1100.7836074829102" disk.dm-0.out‚17019.6201171875 disk.sda.out‚ 122.30859375 disk.dm-1.out‚ 122.30859375¡‹Mÿ2Ø- _wandbruntime‚3032  _runtime‚3032pHk¸ÉÙà _step‚325 ' lr-AdamW/pg1‚0.00019180497199374605 ' lr-AdamW/pg2‚0.00019180497199374605 trainer/global_step‚3999 % _timestamp‚1.7884313479205472e+09 _runtime‚3032.625298117X-AÎÚÈ ' lr-AdamW/pg1‚0.00019180497199374605 ' lr-AdamW/pg2‚0.00019180497199374605 trainer/global_step‚3999 % _timestamp‚1.7884313479205472e+09 _runtime‚3032.625298117 _step‚325ÅÕüÔµ2Û- _wandbruntime‚3032  _runtime‚3032þ ÈâÝ'Ü×O SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3335348561234 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53070874 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚257295239930880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232389925318144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚253782095659520 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232389943321088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3347673460224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133598433 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33906958 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 % _timestamp‚1.7884313479214342e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58113930 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3512950415872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3336002998990 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3347673460224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚232389943321088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75464463 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19164371 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚3032.626489893 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58113932 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235737598778368 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58113930 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚253782119599616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229638746209998 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133578393 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3348329822720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58113932 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75464463 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75484501 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33906956 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3513120331264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53071329 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _step‚326 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235738273143808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58113930 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚253782119599616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229638746209998 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133578393 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3348329822720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58113932 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75464463 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75484501 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33906956 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3513120331264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53071329 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _step‚326 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235738273143808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53070874 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚257295239930880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232389925318144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚253782095659520 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232389943321088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3347673460224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133598433 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33906958 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 % _timestamp‚1.7884313479214342e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58113930 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3512950415872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3336002998990 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3347673460224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚232389943321088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75464463 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19164371 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚3032.626489893 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58113932 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235737598778368 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133630929 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚3999 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58127946 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19168922 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58127948 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232446296359936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75482917 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53084048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33915124 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235795445623808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253843770337280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3513971116032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235794771245056 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133610863 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53083590 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226357647341568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3348492888064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235795445623808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3336819434162 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚232446278356992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3348492888064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3513805408256 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229694466775730 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚3033.522966906 _step‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133630929 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚257357551805440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75502981 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232446278356992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75502981 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 D>DeviceStatsMonitor.ot”T²n_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58127946 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253843746397184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19168466 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226357630302208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229693795298406 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3349149263872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚3033.522966906 _step‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133630929 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚257357551805440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75502981 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232446278356992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75502981 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58127946 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253843746397184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19168466 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226357630302208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229693795298406 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3349149263872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133630929 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚3999 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58127946 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19168922 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58127948 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232446296359936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75482917 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53084048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33915124 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235795445623808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253843770337280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3513971116032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235794771245056 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133610863 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53083590 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226357647341568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3348492888064Ǻ #±2á- _wandbruntime‚3033  _runtime‚3033ö£`Ã]â× &loss/total_loss‚47.806303843848696 0loss/FRt3/assignment_loss‚3.5568275682387815 0loss/FBt4/assignment_loss‚0.7182781406279121 .loss/FRt4/detection_loss‚2.181660852261952 _step‚328 0loss/FRt1/assignment_loss‚1.7646077928634791 /loss/FRt3/detection_loss‚2.7299052323064497 0loss/SRqqt3/detection_loss‚2.613449643765177 % _timestamp‚1.7884313488193855e+09 /loss/FRt2/detection_loss‚2.4233240192396597 /loss/FBt4/detection_loss‚2.3009455033711026 epoch‚3 trainer/global_step‚3999 0loss/FRt2/assignment_loss‚3.2302259781904388 1loss/FBt1/assignment_loss‚0.17649875496910217 /loss/FRt1/detection_loss‚1.7533413593012552 1loss/SRqqt1/detection_loss‚0.8896705418825149 /loss/FBt1/detection_loss‚0.9422536147164868 2loss/SRqqt1/assignment_loss‚1.2243402805711543 0loss/SRqqt4/detection_loss‚2.047995369191523 /loss/FBt2/detection_loss‚2.8696328835053877 /loss/FBt3/detection_loss‚3.0807319432497025 _runtime‚3033.524039657 2loss/SRqqt2/assignment_loss‚1.8622335597872735 2loss/SRqqt4/assignment_loss‚1.3798938528807074 /loss/FBt3/assignment_loss‚2.101396325742826 1loss/SRqqt2/detection_loss‚1.7867166912555694 /loss/FRt4/assignment_loss‚4.214875637268533 2loss/SRqqt3/assignment_loss‚1.8420683156166757 1loss/FBt2/assignment_loss‚0.11542998304503271€ TÀbãÜ 0loss/FRt1/assignment_loss‚1.7646077928634791 /loss/FRt3/detection_loss‚2.7299052323064497 0loss/SRqqt3/detection_loss‚2.613449643765177 % _timestamp‚1.7884313488193855e+09 /loss/FRt2/detection_loss‚2.4233240192396597 /loss/FBt4/detection_loss‚2.3009455033711026 epoch‚3 trainer/global_step‚3999 0loss/FRt2/assignment_loss‚3.2302259781904388 1loss/FBt1/assignment_loss‚0.17649875496910217 /loss/FRt1/detection_loss‚1.7533413593012552 1loss/SRqqt1/detection_loss‚0.8896705418825149 /loss/FBt1/detection_loss‚0.9422536147164868 2loss/SRqqt1/assignment_loss‚1.2243402805711543 0loss/SRqqt4/detection_loss‚2.047995369191523 /loss/FBt2/detection_loss‚2.8696328835053877 /loss/FBt3/detection_loss‚3.0807319432497025 _runtime‚3033.524039657 2loss/SRqqt2/assignment_loss‚1.8622335597872735 2loss/SRqqt4/assignment_loss‚1.3798938528807074 /loss/FBt3/assignment_loss‚2.101396325742826 1loss/SRqqt2/detection_loss‚1.7867166912555694 /loss/FRt4/assignment_loss‚4.214875637268533 2loss/SRqqt3/assignment_loss‚1.8420683156166757 1loss/FBt2/assignment_loss‚0.11542998304503271 &loss/total_loss‚47.806303843848696 0loss/FRt3/assignment_loss‚3.5568275682387815 0loss/FBt4/assignment_loss‚0.7182781406279121 .loss/FRt4/detection_loss‚2.181660852261952 _step‚328È0ËŸÉä‚(:’ €˜åÔë«ð gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚92.998' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.599600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09Fþ˜å‚(: €˜åÔ¸äÝ‘ disk.rbd0.out‚0 proc.memory.rssMB‚2794.5 disk.sda2.out‚0 disk.sda3.in‚0" disk.dm-0.out‚17118.7177734375 disk.sda.out‚ 122.7421875 disk.rbd1.out‚ 1.533203125' disk.nvme0c0n1.out‚17118.7177734375$ memory_percent‚96.63197994232178( proc.memory.availableMB‚ 551.81640625 disk.sda2.in‚0% disk./.usageGB‚1100.7836074829102 disk.sda3.out‚ 122.7421875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd2.in‚0( proc.memory.percent‚17.0562744140625 disk.sda1.in‚0 disk.dm-1.out‚ 122.7421875 network.sent‚ 8.545723e+06 disk.sda.in‚0 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 98.68359375 cpu‚25.3403985527577 proc.cpu.threads‚17) disk./.usagePercent‚73.87451151714407 disk.rbd3.out‚0 disk.dm-0.in‚ 98.68359375 disk.sda1.out‚0 disk.nvme0n1.in‚ 98.68359375 disk.rbd0.in‚0 network.recv‚ 1.106666e+06 disk.rbd2.out‚ 102.96875% disk.nvme0n1.out‚17118.7177734375 disk.rbd1.in‚01›¾Ëžæ‚(:“ ˜åÔ¬·Î gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.773' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.354599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Ùß¾ç‚(:‰ ˜åÔÁÒþ$ memory_percent‚96.63197994232178 disk.dm-1.out‚ 123.3828125 disk.rbd2.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 551.81640625 disk.sda.in‚0$ disk.nvme0c0n1.out‚ 17119.1796875 disk.rbd3.in‚0 network.recv‚ 1.113417e+06% disk./.usageGB‚1100.7836074829102 disk.sda2.out‚0 disk.sda3.out‚ 123.3828125 disk.dm-0.in‚ 98.68359375 network.sent‚ 8.654727e+06 disk.rbd1.out‚ 1.533203125 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.percent‚17.0562744140625 cpu‚25.338918744556388" disk.nvme0n1.out‚ 17119.1796875 disk.rbd1.in‚0 disk.drbd1777.out‚0) disk./.usagePercent‚73.87451151714407 disk.nvme0n1.in‚ 98.68359375 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 123.3828125 disk.sda1.out‚0 proc.memory.rssMB‚2794.5 disk.sda1.in‚0 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 98.68359375 disk.sda2.in‚0 disk.rbd2.out‚ 102.984375 disk.drbd1777.in‚0 disk.dm-0.out‚ 17119.1796875'ÓhÊ2è- _wandbruntime‚3068  _runtime‚3068$ðH“ÇéÁ & lr-AdamW/pg1‚0.0001906171060267872 & lr-AdamW/pg2‚0.0001906171060267872 trainer/global_step‚4049 % _timestamp‚1.7884313840631223e+09 _runtime‚3068.767899262 _step‚329DåÅuÌêÆ % _timestamp‚1.7884313840631223e+09 _runtime‚3068.767899262 _step‚329 & lr-AdamW/pg1‚0.0001906171060267872 & lr-AdamW/pg2‚0.0001906171060267872 trainer/global_step‚4049ÉS¤ñ€2ë- _wandbruntime‚3068  _runtime‚3068œwÏ“Ý'ì×O XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34315358 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235207577260544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3555908018688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235207595263488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58814730 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232424788077406 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3389315628544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229047932699648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53706155 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238596910892032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3376183900642 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596236526592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76390515 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚4049 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚4049 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235207595263488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58814730 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232424788077406 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3389315628544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229047932699648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53706155 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238596910892032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3376183900642 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596236526592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944Ê6:M2î- _wandbruntime‚3069  _runtime‚3069ß${ç&ïáM _step‚331 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238653408993280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76408969 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19395838 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3389478693888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3389478693888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3556761640960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229102853869568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4049 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229102836830208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3390135069696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3556931543040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235263948302336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235263930299392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58828746 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76429033 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53718911 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260483238763520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚256926283280384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3390135069696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34323526 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3377000335606 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135237715 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135257781 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235263948302336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232480508643138 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34323524 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884313847361817e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260483044921344 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238654083372032 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135237715 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58828746 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58828748 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19395387 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76429033 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238654083372032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135257781 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360nÜ-ì&ðæM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135237715 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135257781 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235263948302336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232480508643138 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34323524 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884313847361817e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260483044921344 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238654083372032 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135237715 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58828746 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58828748 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19395387 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76429033 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238654083372032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135257781 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚331 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238653408993280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76408969 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19395838 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3389478693888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3389478693888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3556761640960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229102853869568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4049 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229102836830208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3390135069696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3556931543040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235263948302336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235263930299392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58828746 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76429033 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53718911 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260483238763520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚256926283280384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3390135069696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34323526 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3377000335606ˇFn2ñ- _wandbruntime‚3069  _runtime‚3069õÇ WòÑ 2loss/SRqqt4/assignment_loss‚1.6953990737012787 2loss/SRqqt3/assignment_loss‚1.9290911961218407 /loss/FRt1/detection_loss‚1.8096875651758544 epoch‚3 _runtime‚3069.442539598 /loss/FBt4/detection_loss‚2.4206595288382635 1loss/SRqqt2/assignment_loss‚1.690494440209407 2loss/FBt2/assignment_loss‚0.017419620417058468 .loss/FRt3/detection_loss‚2.940515836700797 /loss/FRt2/assignment_loss‚3.425153635582834 1loss/SRqqt2/detection_loss‚1.8321908500332098 .loss/FBt2/detection_loss‚2.906163454055786 %loss/total_loss‚46.48169550354734 trainer/global_step‚4049 % _timestamp‚1.7884313847379158e+09 1loss/SRqqt1/detection_loss‚1.0483057533561344 0loss/SRqqt4/detection_loss‚1.856447908223844 /loss/FRt2/detection_loss‚2.2488150990234232 .loss/FRt4/detection_loss‚2.271151577884501 1loss/SRqqt3/detection_loss‚2.7729960114791474 _step‚332 0loss/FRt1/assignment_loss‚1.5059648764376738 /loss/FRt3/assignment_loss‚4.635257173329592 0loss/FBt1/assignment_loss‚0.3025810772868101 /loss/FBt1/detection_loss‚0.6101610548074666 -loss/FBt3/detection_loss‚3.26793372631073 0loss/FRt4/assignment_loss‚2.8357407410036433 0loss/FBt3/assignment_loss‚0.1719479584523166 /loss/FBt4/assignment_loss‚1.079189416150459 2loss/SRqqt1/assignment_loss‚1.2084279289652764áç~Ö\óÖ 2loss/SRqqt3/assignment_loss‚1.9290911961218407 /loss/FRt1/detection_loss‚1.8096875651758544 epoch‚3 _runtime‚3069.442539598 /loss/FBt4/detection_loss‚2.4206595288382635 1loss/SRqqt2/assignment_loss‚1.690494440209407 2loss/FBt2/assignment_loss‚0.017419620417058468 .loss/FRt3/detection_loss‚2.940515836700797 /loss/FRt2/assignment_loss‚3.425153635582834 1loss/SRqqt2/detection_loss‚1.8321908500332098 .loss/FBt2/detection_loss‚2.906163454055786 %loss/total_loss‚46.48169550354734 trainer/global_step‚4049 % _timestamp‚1.7884313847379158e+09 1loss/SRqqt1/detection_loss‚1.0483057533561344 0loss/SRqqt4/detection_loss‚1.856447908223844 /loss/FRt2/detection_loss‚2.2488150990234232 .loss/FRt4/detection_loss‚2.271151577884501 1loss/SRqqt3/detection_loss‚2.7729960114791474 _step‚332 0loss/FRt1/assignment_loss‚1.5059648764376738 /loss/FRt3/assignment_loss‚4.635257173329592 0loss/FBt1/assignment_loss‚0.3025810772868101 /loss/FBt1/detection_loss‚0.6101610548074666 -loss/FBt3/detection_loss‚3.26793372631073 0loss/FRt4/assignment_loss‚2.8357407410036433 0loss/FBt3/assignment_loss‚0.1719479584523166 /loss/FBt4/assignment_loss‚1.079189416150459 2loss/SRqqt1/assignment_loss‚1.2084279289652764 2loss/SRqqt4/assignment_loss‚1.6953990737012787ÌTáEq‘ô‚(:† ž˜åÔ¼ùì gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ß^[õ‚(:“ ž˜åÔà­Ê disk.sda3.out‚ 123.89453125% disk./.usageGB‚1100.7836647033691 disk.nvme0n1.in‚ 98.68359375 proc.cpu.threads‚17( disk.nvme0c0n1.out‚17218.43408203125& proc.memory.availableMB‚ 551.546875 network.sent‚ 8.659917e+06 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0# disk.dm-0.out‚17218.43408203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚103.125& disk.nvme0n1.out‚17218.43408203125 disk.rbd1.out‚ 1.533203125 cpu‚25.337527967315253 disk.dm-0.in‚ 98.68359375 disk.dm-1.out‚ 123.89453125 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0$ memory_percent‚96.63362503051758 proc.memory.rssMB‚2794.5 network.recv‚ 1.117248e+06) disk./.usagePercent‚73.87451535725708 disk.sda.in‚0 disk.rbd2.in‚0( proc.memory.percent‚17.0562744140625 disk.sda.out‚ 123.89453125 disk.dm-1.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 98.68359375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0ûÈu®“ö‚(:ˆ ­˜åÔχË gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚141.569' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3138 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 î:ð#÷‚(:˜ ­˜åÔé¯ù disk.sda3.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 124.60546875& disk.nvme0n1.out‚17318.07275390625 disk.sda2.in‚0 disk.sda3.out‚ 124.60546875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 98.68359375 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 124.60546875! disk.nvme0c0n1.in‚ 98.68359375 network.recv‚ 1.123465e+06) disk./.usagePercent‚73.87451586927214 disk.dm-0.in‚ 98.68359375 disk.sda.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚ 103.140625# disk.dm-0.out‚17318.07275390625 disk.rbd0.out‚0$ memory_percent‚96.63145542144775 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125( proc.memory.availableMB‚ 551.90234375 network.sent‚ 8.768725e+06( disk.nvme0c0n1.out‚17318.07275390625 proc.memory.rssMB‚2794.5% disk./.usageGB‚1100.7836723327637 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.percent‚17.0562744140625 cpu‚25.336074322479046-ÒÚ>2ø- _wandbruntime‚3104  _runtime‚3104ºDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259947169425920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136852137 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚241454874274816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚238025247205888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54341191 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263546092050432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚241455548640256 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚3104.421431921 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚238025247205888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77336605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77316567 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3430301434368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚241455548640256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3430301434368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚241454874274816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3429645071872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚238025229202944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59515532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3417019240050 _step‚334 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231793139227648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884314197165356e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3429645071872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34723758 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235210829944814 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3417673677806 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77336605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231793156267008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259947169425920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136852137 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚241454874274816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚238025247205888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54341191 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263546092050432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚241455548640256 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚3104.421431921 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚238025247205888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77336605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77316567 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3430301434368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚241455548640256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3430301434368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚241454874274816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3429645071872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚238025229202944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59515532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3417019240050 _step‚334 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231793139227648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884314197165356e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3429645071872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34723758 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235210829944814 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3417673677806 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77336605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231793156267008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3431120875520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260008820163584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3418490112978 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136864567 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54353508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19622025 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884314203873093e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231848060397568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚238081582241792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3599752042496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260008844103680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚238081600244736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136864567 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4099 _runtime‚3105.092419734 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77355085 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19621584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235266550510546 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3430464499712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34731924 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136884633 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77355085 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3430464499712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136884633 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚241512721120256 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59529548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202230784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235265879033222 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚335 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3599573751808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34731926 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928•VYc‚æM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77335021 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3431120875520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260008820163584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3418490112978 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.smGÄ9=‰!all_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136864567 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54353508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19622025 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884314203873093e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231848060397568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚238081582241792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3599752042496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260008844103680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚238081600244736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136864567 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4099 _runtime‚3105.092419734 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77355085 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19621584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235266550510546 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3430464499712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34731924 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136884633 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77355085 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3430464499712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136884633 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚241512721120256 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59529548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202230784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235265879033222 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚335 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3599573751808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34731926 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59529548 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360Ϻ6¨‡2ƒ- _wandbruntime‚3105  _runtime‚3105Κ~JV„Ð 0loss/SRqqt3/assignment_loss‚1.84484792859466 0loss/SRqqt1/detection_loss‚1.158163408006447 1loss/SRqqt2/detection_loss‚1.6519428004643746 _step‚336 /loss/FRt3/detection_loss‚3.1120296076492027 epoch‚3 % _timestamp‚1.7884314203888292e+09 2loss/SRqqt1/assignment_loss‚1.0762946852415367 0loss/FBt2/assignment_loss‚0.2588192032650113 0loss/FBt4/assignment_loss‚0.6283822340192273 -loss/FRt1/detection_loss‚2.08559018433094 /loss/FRt2/detection_loss‚2.3975890376351097 /loss/FRt4/detection_loss‚2.2372749732299284 1loss/FBt3/assignment_loss‚0.16762113259043568 /loss/FBt2/detection_loss‚3.0414400696754456 /loss/FBt3/detection_loss‚3.6173469920953116 .loss/FBt4/detection_loss‚2.418062426149845 1loss/FBt1/assignment_loss‚0.17811459543180597 1loss/SRqqt3/detection_loss‚2.7229539078694804 0loss/SRqqt4/detection_loss‚2.115898147970438 &loss/total_loss‚46.994931762656535 trainer/global_step‚4099 _runtime‚3105.093442261 0loss/FRt1/assignment_loss‚1.2565793108940124 /loss/FRt2/assignment_loss‚3.189143279465762 2loss/SRqqt2/assignment_loss‚1.4738939882788276 /loss/FRt3/assignment_loss‚4.184966105001944 /loss/FRt4/assignment_loss‚3.795186535878615 .loss/FBt1/detection_loss‚0.714556878409312 1loss/SRqqt4/assignment_loss‚1.668234330508858ð‚C[…Õ /loss/FRt3/assignment_loss‚4.184966105001944 /loss/FRt4/assignment_loss‚3.795186535878615 .loss/FBt1/detection_loss‚0.714556878409312 1loss/SRqqt4/assignment_loss‚1.668234330508858 0loss/SRqqt3/assignment_loss‚1.84484792859466 0loss/SRqqt1/detection_loss‚1.158163408006447 1loss/SRqqt2/detection_loss‚1.6519428004643746 _step‚336 /loss/FRt3/detection_loss‚3.1120296076492027 epoch‚3 % _timestamp‚1.7884314203888292e+09 2loss/SRqqt1/assignment_loss‚1.0762946852415367 0loss/FBt2/assignment_loss‚0.2588192032650113 0loss/FBt4/assignment_loss‚0.6283822340192273 -loss/FRt1/detection_loss‚2.08559018433094 /loss/FRt2/detection_loss‚2.3975890376351097 /loss/FRt4/detection_loss‚2.2372749732299284 1loss/FBt3/assignment_loss‚0.16762113259043568 /loss/FBt2/detection_loss‚3.0414400696754456 /loss/FBt3/detection_loss‚3.6173469920953116 .loss/FBt4/detection_loss‚2.418062426149845 1loss/FBt1/assignment_loss‚0.17811459543180597 1loss/SRqqt3/detection_loss‚2.7229539078694804 0loss/SRqqt4/detection_loss‚2.115898147970438 &loss/total_loss‚46.994931762656535 trainer/global_step‚4099 _runtime‚3105.093442261 0loss/FRt1/assignment_loss‚1.2565793108940124 /loss/FRt2/assignment_loss‚3.189143279465762 2loss/SRqqt2/assignment_loss‚1.4738939882788276Ð÷ÁFœ†‚(:‘ ˘åÔ“¹ÿ gpu.0.gpu‚24.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.29' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.458000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øvÔ$‡‚(:™ ˘åÔ‰ðÝ" disk.nvme0c0n1.in‚ 100.18359375( disk.nvme0c0n1.out‚17517.28564453125 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0) disk./.usagePercent‚73.87451279718174 disk.dm-1.out‚ 125.66015625 disk.rbd1.in‚0 disk.sda2.in‚0 cpu‚25.33333312517948 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.533203125 disk.rbd2.in‚0$ memory_percent‚96.63186073303223' proc.memory.availableMB‚ 551.8359375( proc.memory.percent‚17.0562744140625 network.recv‚ 1.13358e+06& disk.nvme0n1.out‚17517.28564453125 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 100.18359375 disk.sda.in‚0 disk.sda.out‚ 125.66015625 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 125.66015625 disk.nvme0n1.in‚ 100.18359375 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 104.79296875 proc.memory.rssMB‚2794.5 network.sent‚ 8.87162e+06% disk./.usageGB‚1100.7836265563965# disk.dm-0.out‚17517.28564453125 disk.sda1.in‚0“Et“ˆ‚(:ˆ Ú˜åÔ‰ò gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.044' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.8088 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ë "‰‚(:— Ú˜åÔʆó disk.dm-1.in‚0 proc.memory.rssMB‚2794.5% disk./.usageGB‚1100.7836074829102 disk.sda1.out‚0& disk.nvme0n1.out‚17517.55517578125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 126.19921875( proc.memory.availableMB‚ 551.68359375 network.recv‚ 1.138754e+06# disk.dm-0.out‚17517.55517578125 disk.nvme0n1.in‚ 100.34375( disk.nvme0c0n1.out‚17517.55517578125$ memory_percent‚96.63279056549072( proc.memory.percent‚17.0562744140625 cpu‚25.332065125948176 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.nvme0c0n1.in‚ 100.34375 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0! disk.rbd2.out‚104.87939453125 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 126.19921875 disk.rbd1.out‚ 1.533203125 disk.dm-0.in‚ 100.34375 disk.sda.out‚ 126.19921875 disk.sda2.out‚0 network.sent‚ 8.969017e+06) disk./.usagePercent‚73.87451151714407WÔ¤#2Š- _wandbruntime‚3139  _runtime‚3139Až¼É‹Ã _runtime‚3139.741073109 _step‚337 ' lr-AdamW/pg1‚0.00018819329382203503 ' lr-AdamW/pg2‚0.00018819329382203503 trainer/global_step‚4149 % _timestamp‚1.7884314550363345e+09ÉmÎŒÈ % _timestamp‚1.7884314550363345e+09 _runtime‚3139.741073109 _step‚337 ' lr-AdamW/pg1‚0.00018819329382203503 ' lr-AdamW/pg2‚0.00018819329382203503 trainer/global_step‚4149ѬÎ92- _wandbruntime‚3139  _runtime‚3139·ß*nÝ'Ž×O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240842881145344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3470630877696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263029730249216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3458509017214 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266671518796800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚4149 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237996200335106 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚240842881145344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266671335427072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234538362795008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244313512023040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19842811 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60216332 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚338 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244314186388480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3471287240192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78262657 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3641629117952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3139.742239367 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3641788547584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237996871812222 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234538345755648 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60216330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35132158 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3457854579458 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚244314186388480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244313512023040 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60216332 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35132156 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54974969 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78242619 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19842382 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884314550372372e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138458949 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240842899148288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3470630877696 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78262657 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3641629117952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3139.742239367 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3641788547584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237996871812222 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234538345755648 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60216330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35132158 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3457854579458 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚244314186388480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244313512023040 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60216332 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35132156 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54974969 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78242619 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19842382 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884314550372372e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138458949 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240842899148288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3470630877696 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240842881145344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3470630877696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263029730249216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3458509017214 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266671518796800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚4149 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237996200335106 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚240842881145344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266671335427072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pooÑ$_ý l.allocated‚234538362795008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244313512023040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19842811 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60216332 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚338 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244314186388480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3471287240192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _step‚339 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78281137 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚238051920900630 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244371358868480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19846944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238052592377954 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚4149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35140326 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884314556878433e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚234593266925568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54987268 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚ 3140.39327315 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244371358868480 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54987686 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266733871553536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 146833408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240899252187136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263091380986880 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3642514506752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78261073 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78261073 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138511485 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60230348 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚418 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3472106681344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3471450305536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3471450305536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3459325452386 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60230346 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240899252187136 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54987268 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚ 3140.39327315 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244371358868480 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54987686 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266733871553536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 146833408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240899252187136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263091380986880 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3642514506752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78261073 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78261073 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138511485 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60230348 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚418 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3472106681344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3471450305536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3471450305536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3459325452386 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60230346 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240899252187136 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _step‚339 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401Óèb22“- _wandbruntime‚3140  _runtime‚3140ýklêW”Ñ /loss/FBt3/assignment_loss‚0.829928519802609 0loss/FRt2/assignment_loss‚2.8870758593082426 /loss/FRt1/detection_loss‚1.6147032039096723 1loss/SRqqt3/assignment_loss‚1.881925566838338 0loss/FBt4/assignment_loss‚0.4305015314370394 1loss/SRqqt4/detection_loss‚1.7381008985367687 .loss/FRt2/detection_loss‚2.173816530406475 0loss/FBt2/assignment_loss‚0.1587359273340553 1loss/SRqqt1/detection_loss‚1.1194287961654699 /loss/FBt2/detection_loss‚2.5274502992630006 .loss/FRt4/detection_loss‚2.229105743707395 /loss/FBt1/detection_loss‚0.6604249873245096 /loss/FBt4/detection_loss‚2.5791029155254366 epoch‚3 trainer/global_step‚4149 % _timestamp‚1.7884314556895828e+09 _step‚340 0loss/FRt4/assignment_loss‚3.5413354565115536 2loss/SRqqt2/assignment_loss‚1.4149844086108108 2loss/SRqqt4/assignment_loss‚1.3219908839938315 /loss/FRt3/detection_loss‚3.0025806676956917 .loss/FBt3/detection_loss‚2.714104541710445 _runtime‚3140.394234061 0loss/FRt1/assignment_loss‚1.5399737002757878 0loss/SRqqt2/detection_loss‚1.948036809451878 0loss/SRqqt3/detection_loss‚2.820979912693684 &loss/total_loss‚44.444113314715395 1loss/SRqqt1/assignment_loss‚1.128024631582979 0loss/FBt1/assignment_loss‚0.2965442323740833 /loss/FRt3/assignment_loss‚3.885257290255639mþß\•Ö % _timestamp‚1.7884314556895828e+09 _step‚340 0loss/FRt4/assignment_loss‚3.5413354565115536 2loss/SRqqt2/assignment_loss‚1.4149844086108108 2loss/SRqqt4/assignment_loss‚1.3219908839938315 /loss/FRt3/detection_loss‚3.0025806676956917 .loss/FBt3/detection_loss‚2.714104541710445 _runtime‚3140.394234061 0loss/FRt1/assignment_loss‚1.5399737002757878 0loss/SRqqt2/detection_loss‚1.948036809451878 0loss/SRqqt3/detection_loss‚2.820979912693684 &loss/total_loss‚44.444113314715395 1loss/SRqqt1/assignment_loss‚1.128024631582979 0loss/FBt1/assignment_loss‚0.2965442323740833 /loss/FRt3/assignment_loss‚3.885257290255639 /loss/FBt3/assignment_loss‚0.829928519802609 0loss/FRt2/assignment_loss‚2.8870758593082426 /loss/FRt1/detection_loss‚1.6147032039096723 1loss/SRqqt3/assignment_loss‚1.881925566838338 0loss/FBt4/assignment_loss‚0.4305015314370394 1loss/SRqqt4/detection_loss‚1.7381008985367687 .loss/FRt2/detection_loss‚2.173816530406475 0loss/FBt2/assignment_loss‚0.1587359273340553 1loss/SRqqt1/detection_loss‚1.1194287961654699 /loss/FBt2/detection_loss‚2.5274502992630006 .loss/FRt4/detection_loss‚2.229105743707395 /loss/FBt1/detection_loss‚0.6604249873245096 /loss/FBt4/detection_loss‚2.5791029155254366 epoch‚3 trainer/global_step‚4149Ô²CFbž–‚(:“ é˜åÔÿ¯‚Ž gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.802' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.760399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ]}—‚(: é˜åÔïµ disk.rbd3.out‚0 disk.sda.out‚ 126.7890625$ memory_percent‚96.63257598876953 proc.memory.rssMB‚2794.5% proc.memory.availableMB‚ 551.71875 cpu‚25.33066201107964) disk./.usagePercent‚73.87449564467694 disk.sda3.in‚0 disk.sda3.out‚ 126.7890625 disk.dm-1.in‚0 disk.dm-1.out‚ 126.7890625 disk.nvme0n1.in‚ 100.42578125$ disk.nvme0c0n1.out‚ 17616.8515625! disk.rbd2.out‚104.87939453125% disk./.usageGB‚1100.7833709716797 disk.sda2.in‚0( proc.memory.percent‚17.0562744140625 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.619140625 disk.rbd2.in‚0 network.recv‚ 1.143097e+06 disk.drbd1777.out‚0" disk.nvme0n1.out‚ 17616.8515625 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 100.42578125 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 17616.8515625 disk.sda.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 100.42578125 network.sent‚ 8.975306e+06 disk.rbd3.in‚0ö V•ž˜‚(:“ ø˜åÔ–ü‡Ž gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.274800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O[#ã,™‚(:¡ ø˜åÔüê  disk.sda1.out‚0& disk.nvme0n1.out‚17716.00146484375 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 100.42578125 disk.sda2.out‚0! proc.memory.rssMB‚ 2794.515625) disk./.usagePercent‚73.87449436463928 disk.sda3.out‚ 127.4609375 disk.sda.out‚ 127.4609375 disk.nvme0n1.in‚ 100.42578125( disk.nvme0c0n1.out‚17716.00146484375 disk.rbd3.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.rbd1.out‚ 1.681640625 disk.rbd0.out‚0$ memory_percent‚96.63329124450684 network.recv‚ 1.149321e+06' proc.memory.availableMB‚ 551.6015625 cpu‚25.32934536975253 proc.cpu.threads‚17 network.sent‚ 9.083398e+06 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 100.42578125# disk.dm-0.out‚17716.00146484375 disk.dm-1.out‚ 127.4609375) proc.memory.percent‚17.05636978149414% disk./.usageGB‚1100.7833518981934 disk.rbd2.in‚0! disk.rbd2.out‚105.02001953125U¸2š- _wandbruntime‚3175  _runtime‚31750¥2ɛà % _timestamp‚1.7884314905314908e+09 _runtime‚3175.236235803 _step‚341 ' lr-AdamW/pg1‚0.00018695785270307234 ' lr-AdamW/pg2‚0.00018695785270307234 trainer/global_step‚4199:ÔÒÎœÈ ' lr-AdamW/pg1‚0.00018695785270307234 ' lr-AdamW/pg2‚0.00018695785270307234 trainer/global_step‚4199 % _timestamp‚1.7884314905314908e+09 _runtime‚3175.236235803 _step‚341Õ®yd 2- _wandbruntime‚3175  _runtime‚3175)KÆ0žÖO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚247172149771264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60917130 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243660533087744 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35540558 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35540556 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚266112267132416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243660551090688 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266112243192320 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚4199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240782913679630 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3512273046016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140105841 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79168671 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20068109 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55609125 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60917132 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3498689918866 _step‚342 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3511616683520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204341248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3684386036224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79188709 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3512273046016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243660551090688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3511616683520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20068567 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788431490532194e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269796833569792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140085801 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79168671 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269796629228544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20068567 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788431490532194e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269796833569792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140085801 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79168671 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269796629228544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚247172149771264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60917130 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243660533087744 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35540558 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35540556 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚266112267132416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243660551090688 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266112243192320 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚4199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240782913679630 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3512273046016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140105841 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79168671 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20068109 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414Öêï™2 - _wandbruntime‚3175  _runtime‚3175‹KÓ)ç&¡áM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140138337 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3513092487168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240838634245362 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60931148 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55621325 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3685415026688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3685251416064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243716904129536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60931148 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243716886126592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3512436111360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237338456414208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35548726 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79207189 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3500160791794 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚247229996616704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚243716904129536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 % _timestamp‚1.7884314911867466e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3513092487168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140138337 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237338473453568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚343 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20072601 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚4199 _runtime‚3175.892095777 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79187125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20073030 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79187125 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55621756Î\éÑ7¢æM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140138337 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3513092487168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240838634245362 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60931148 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55621325 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3685415026688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3685251416064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243716904129536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60931148 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243716886126592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3512436111360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237338456414208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35548726 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79207189 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3500160791794 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚247229996616704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚243716904129536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 % _timestamp‚1.7884314911867466e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3513092487168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140138337 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 ^JDeviceStatsMonitor.on_train_batch_end/requested_byoqaSµtes.large_pool.allocated‚237338473453568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚343 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20072601 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚4199 _runtime‚3175.892095777 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79187125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20073030 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79187125 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55621756מ˜K2£- _wandbruntime‚3175  _runtime‚3175Tj$£[¤Õ 2loss/SRqqt4/assignment_loss‚1.5136041096515125 epoch‚3 _step‚344 1loss/FBt1/assignment_loss‚0.22917100026444764 -loss/FRt2/assignment_loss‚2.7301036241727 1loss/FBt4/assignment_loss‚0.21648297063075006 /loss/FBt4/detection_loss‚2.8472767025232315 &loss/total_loss‚43.432759408864186 trainer/global_step‚4199 /loss/FRt3/assignment_loss‚3.356387874111533 1loss/FBt3/assignment_loss‚0.27723422574345025 /loss/FRt4/detection_loss‚2.0242917849258943 1loss/SRqqt1/detection_loss‚0.9950088270369076 1loss/SRqqt4/detection_loss‚1.8499826259083219 .loss/FBt2/detection_loss‚2.555876672267914 2loss/SRqqt1/assignment_loss‚0.9226406503528837 1loss/FBt2/assignment_loss‚0.17164734642137774 0loss/SRqqt2/detection_loss‚1.918192987525186 1loss/SRqqt3/detection_loss‚2.5790005470767166 /loss/FRt4/assignment_loss‚3.315644539215348 1loss/SRqqt3/assignment_loss‚1.902953075188579 /loss/FRt3/detection_loss‚3.0341008827090263 2loss/SRqqt2/assignment_loss‚1.3105307055074116 /loss/FRt1/detection_loss‚1.7843171115568344 % _timestamp‚1.7884314911884599e+09 0loss/FRt1/assignment_loss‚1.6763115499881989 .loss/FRt2/detection_loss‚2.440308479926525 /loss/FBt1/detection_loss‚0.5788582921676588 /loss/FBt3/detection_loss‚3.2028328239917756 _runtime‚3175.893105841ƒ¨h8`¥Ú 0loss/FRt1/assignment_loss‚1.6763115499881989 .loss/FRt2/detection_loss‚2.440308479926525 /loss/FBt1/detection_loss‚0.5788582921676588 /loss/FBt3/detection_loss‚3.2028328239917756 _runtime‚3175.893105841 2loss/SRqqt4/assignment_loss‚1.5136041096515125 epoch‚3 _step‚344 1loss/FBt1/assignment_loss‚0.22917100026444764 -loss/FRt2/assignment_loss‚2.7301036241727 1loss/FBt4/assignment_loss‚0.21648297063075006 /loss/FBt4/detection_loss‚2.8472767025232315 &loss/total_loss‚43.432759408864186 trainer/global_step‚4199 /loss/FRt3/assignment_loss‚3.356387874111533 1loss/FBt3/assignment_loss‚0.27723422574345025 /loss/FRt4/detection_loss‚2.0242917849258943 1loss/SRqqt1/detection_loss‚0.9950088270369076 1loss/SRqqt4/detection_loss‚1.8499826259083219 .loss/FBt2/detection_loss‚2.555876672267914 2loss/SRqqt1/assignment_loss‚0.9226406503528837 1loss/FBt2/assignment_loss‚0.17164734642137774 0loss/SRqqt2/detection_loss‚1.918192987525186 1loss/SRqqt3/detection_loss‚2.5790005470767166 /loss/FRt4/assignment_loss‚3.315644539215348 1loss/SRqqt3/assignment_loss‚1.902953075188579 /loss/FRt3/detection_loss‚3.0341008827090263 2loss/SRqqt2/assignment_loss‚1.3105307055074116 /loss/FRt1/detection_loss‚1.7843171115568344 % _timestamp‚1.7884314911884599e+09Ø•:A ž¦‚(:“ ‡™åÔÿ’í gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.115' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.423000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gÓü”#§‚(:˜ ‡™åÔš‚ disk.nvme0n1.in‚ 100.48828125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 100.48828125 disk.dm-0.out‚17815.50390625 disk.dm-1.out‚ 128.078125 disk.sda.out‚ 128.078125) disk./.usagePercent‚73.87449436463928# disk.nvme0n1.out‚17815.50390625" disk.nvme0c0n1.in‚ 100.48828125 disk.rbd3.out‚0 disk.sda.in‚0 cpu‚25.32795616341057 disk.rbd1.in‚0 disk.rbd0.in‚0! disk.rbd2.out‚106.59423828125 disk.sda1.out‚0% disk./.usageGB‚1100.7833518981934 disk.drbd1777.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17$ memory_percent‚96.63465023040771" proc.memory.rssMB‚ 2794.5703125 disk.sda1.in‚0 network.sent‚ 9.089673e+06 disk.sda3.out‚ 128.078125 disk.drbd1777.out‚0( proc.memory.availableMB‚ 551.37890625 network.recv‚ 1.153752e+06% disk.nvme0c0n1.out‚17815.50390625 disk.dm-1.in‚0* proc.memory.percent‚17.056703567504883 disk.rbd1.out‚ 1.681640625 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0|©³Z“¨‚(:ˆ –™åԶ딎 gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ñï©‚(:Ž –™åÔÂç·! disk.nvme0c0n1.out‚ 17815.8125 disk.dm-1.in‚0 disk.dm-1.out‚ 128.7109375 disk.rbd1.out‚ 1.7138671875 disk.sda2.out‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2794.5703125* proc.memory.percent‚17.056703567504883 proc.cpu.threads‚17 disk.dm-0.in‚ 102.4921875 disk.rbd0.out‚0 disk.drbd1777.in‚0 network.sent‚ 9.197471e+06 disk.dm-0.out‚ 17815.8125 disk.sda.in‚0 disk.nvme0n1.in‚ 102.4921875! disk.rbd2.out‚106.67236328125 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 551.99609375 cpu‚25.326663037720852 disk.sda1.in‚0 disk.rbd1.in‚0 network.recv‚ 1.159182e+06% disk./.usageGB‚1100.7836723327637 disk.sda3.out‚ 128.7109375$ memory_percent‚96.63088321685791 disk.nvme0n1.out‚ 17815.8125! disk.nvme0c0n1.in‚ 102.4921875) disk./.usagePercent‚73.87451586927214 disk.sda.out‚ 128.7109375 disk.sda1.out‚0‘íÍ“ª‚(:ˆ ¥™åÔÆõâ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.909' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.1818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Húà5+«‚(:  ¥™åÔþé¡% proc.memory.availableMB‚ 550.78125 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 network.sent‚ 9.203751e+06 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0# disk.dm-0.out‚17914.98974609375 disk.dm-1.out‚ 129.546875 disk.rbd1.in‚0 disk.sda3.out‚ 129.546875 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda.out‚ 129.546875" proc.memory.rssMB‚ 2794.5703125) disk./.usagePercent‚73.87451458923448! disk.rbd2.out‚106.67236328125 disk.rbd3.out‚0 cpu‚25.325460646193072 proc.cpu.threads‚17 network.recv‚ 1.163699e+06% disk./.usageGB‚1100.7836532592773 disk.rbd0.out‚0 disk.rbd1.out‚ 1.7138671875( disk.nvme0c0n1.out‚17914.98974609375 disk.rbd2.in‚0$ memory_percent‚96.63829803466797* proc.memory.percent‚17.056703567504883 disk.sda1.in‚0 disk.nvme0n1.in‚ 103.92578125& disk.nvme0n1.out‚17914.98974609375" disk.nvme0c0n1.in‚ 103.92578125 disk.rbd0.in‚0 disk.dm-0.in‚ 103.92578125²¡402¬- _wandbruntime‚3210  _runtime‚3210‘Á¿É­Ã _runtime‚3210.793305117 _step‚345 ' lr-AdamW/pg1‚0.00018570706377444998 ' lr-AdamW/pg2‚0.00018570706377444998 trainer/global_step‚4249 % _timestamp‚1.7884315260885327e+09&Ú¨!Î®È _runtime‚3210.793305117 _step‚345 ' lr-AdamW/pg1‚0.00018570706377444998 ' lr-AdamW/pg2‚0.00018570706377444998 trainer/global_step‚4249 % _timestamp‚1.7884315260885327e+09ÙÆÖ­z2¯- _wandbruntime‚3210  _runtime‚3210•("ìÜ'°ÖO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727418882560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56243626 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚246478203033088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61617932 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61617930 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚246478185030144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20294220 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269194780075520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80114761 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3553258851840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272922222898176 trainer/global_step‚4249 % _timestamp‚1.7884315260892918e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3552602489344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80094723 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250030787519488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3552602489344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3727251064320 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250030787519488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240028775851008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141732693 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243568955547038 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚246478185030144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 3210.79419302 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20294668 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243568284069922 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚246478203033088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240028758811648 _step‚346 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272922031139840 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80094723 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336¶Èçá'±ÛO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272922031139840 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80094723 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727418882560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56243626 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚246478203033088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61617932 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61617930 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚246478185030144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20294220 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269194780075520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80114761 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3553258851840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272922222898176 trainer/global_step‚4249 % _timestamp‚1.7884315260892918e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3552602489344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80094723 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250030787519488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3552602489344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3727251064320 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250030787519488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240028775851008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141732693 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243568955547038 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚246478185030144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 3210.79419302 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20294668 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243568284069922 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚246478203033088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240028758811648 _step‚346 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Úéõ>Ž2²- _wandbruntime‚3211  _runtime‚3211»†ú Ë³àM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61631948 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytesŸìÿÝ$.small_pool.freed‚ 3553421917184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141745123 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚246534556071936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61631946 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3728271551488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3540341693238 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141765189 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80133241 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3553421917184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35957126 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80113177 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚250088634364928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80133241 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚269256454753280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56256157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56255713 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246534538068992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141765189 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚269256430813184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚3211.454760376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 $ _timestamp‚1.788431526749266e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚246534556071936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚250088634364928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4249 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20299031 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3540996131202 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚347 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚3211.454760376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 $ _timestamp‚1.788431526749266e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚246534556071936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚250088634364928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4249 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20299031 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3540996131202 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚347 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61631948 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3553421917184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141745123 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚246534556071936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61631946 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3728271551488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3540341693238 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141765189 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80133241 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3553421917184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35957126 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80113177 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚250088634364928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80133241 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚269256454753280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56256157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56255713 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246534538068992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680Û4T”2µ- _wandbruntime‚3211  _runtime‚3211M8íY¶Ó 2loss/SRqqt2/assignment_loss‚1.7748510440618828 .loss/FRt1/detection_loss‚1.943930946036496 1loss/SRqqt4/detection_loss‚1.8272140706362932 /loss/FRt1/assignment_loss‚1.621615010229024 1loss/FBt4/assignment_loss‚0.32135966089036727 _runtime‚3211.456084091 2loss/SRqqt1/assignment_loss‚0.7193130676544481 1loss/SRqqt3/detection_loss‚2.4265984892845154 epoch‚3 0loss/FBt1/assignment_loss‚0.0858502131939021 .loss/FRt2/detection_loss‚2.398573169924996 /loss/FBt1/detection_loss‚0.5681574359647081 /loss/FBt2/detection_loss‚2.9273357689380646 %loss/total_loss‚44.79569719974021 0loss/FRt2/assignment_loss‚3.5501582524993203 /loss/FRt4/assignment_loss‚3.986952101406844 -loss/FRt3/detection_loss‚2.99414069693664 /loss/FBt4/detection_loss‚2.6548828217718334 trainer/global_step‚4249 _step‚348 0loss/FRt3/assignment_loss‚2.8180599459286393 1loss/SRqqt3/assignment_loss‚2.019210413680412 1loss/FBt2/assignment_loss‚0.14969465807371307 0loss/FBt3/assignment_loss‚0.2940776660107076 1loss/SRqqt2/detection_loss‚1.6015493993101448 .loss/FBt3/detection_loss‚3.374386042356491 2loss/SRqqt4/assignment_loss‚1.2178314742024825 /loss/FRt4/detection_loss‚2.2862323030181555 1loss/SRqqt1/detection_loss‚1.2337225477301306 % _timestamp‚1.7884315267514255e+09uA’Ž^·Ø 0loss/FRt2/assignment_loss‚3.5501582524993203 /loss/FRt4/assignment_loss‚3.986952101406844 -loss/FRt3/detection_loss‚2.99414069693664 /loss/FBt4/detection_loss‚2.6548828217718334 trainer/global_step‚4249 _step‚348 0loss/FRt3/assignment_loss‚2.8180599459286393 1loss/SRqqt3/assignment_loss‚2.019210413680412 1loss/FBt2/assignment_loss‚0.14969465807371307 0loss/FBt3/assignment_loss‚0.2940776660107076 1loss/SRqqt2/detection_loss‚1.6015493993101448 .loss/FBt3/detection_loss‚3.374386042356491 2loss/SRqqt4/assignment_loss‚1.2178314742024825 /loss/FRt4/detection_loss‚2.2862323030181555 1loss/SRqqt1/detection_loss‚1.2337225477301306 % _timestamp‚1.7884315267514255e+09 2loss/SRqqt2/assignment_loss‚1.7748510440618828 .loss/FRt1/detection_loss‚1.943930946036496 1loss/SRqqt4/detection_loss‚1.8272140706362932 /loss/FRt1/assignment_loss‚1.621615010229024 1loss/FBt4/assignment_loss‚0.32135966089036727 _runtime‚3211.456084091 2loss/SRqqt1/assignment_loss‚0.7193130676544481 1loss/SRqqt3/detection_loss‚2.4265984892845154 epoch‚3 0loss/FBt1/assignment_loss‚0.0858502131939021 .loss/FRt2/detection_loss‚2.398573169924996 /loss/FBt1/detection_loss‚0.5681574359647081 /loss/FBt2/detection_loss‚2.9273357689380646 %loss/total_loss‚44.79569719974021Üßi™tž¸‚(:“ ´™åÔéÄÊ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.153' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.830599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ɺ)&$¹‚(:™ ´™åÔîž× network.recv‚ 1.167056e+06 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2794.5703125) disk./.usagePercent‚73.87451612527968 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.77685546875 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0! disk.dm-0.out‚18014.146484375 disk.nvme0n1.in‚ 112.66015625$ disk./.usageGB‚1100.783676147461 disk.sda.in‚0 disk.dm-1.out‚ 130.05078125& disk.nvme0c0n1.out‚18014.146484375# memory_percent‚96.6339111328125 disk.dm-0.in‚ 112.66015625 disk.sda.out‚ 130.05078125$ disk.nvme0n1.out‚18014.146484375 disk.rbd2.in‚0 cpu‚25.32418311751211 network.sent‚ 9.209148e+06 disk.sda3.in‚0 disk.sda3.out‚ 130.05078125 disk.rbd0.in‚0 disk.rbd2.out‚106.7744140625* proc.memory.percent‚17.056703567504883 disk.drbd1777.out‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 112.66015625! proc.memory.availableMB‚551.5‡Õð÷“º‚(:ˆ ÙåÔÜ÷©Ž gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.6678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h»\»&»‚(:› ÙåÔò鈑" proc.memory.rssMB‚ 2794.5703125 disk.rbd1.out‚ 1.77685546875 disk.rbd0.out‚0) disk./.usagePercent‚73.87452252546804% disk./.usageGB‚1100.7837715148926 disk.sda3.out‚ 130.640625 disk.dm-1.in‚0 disk.dm-1.out‚ 130.640625 disk.sda3.in‚0& disk.nvme0n1.out‚18113.60693359375 network.sent‚ 9.317496e+06# disk.dm-0.out‚18113.60693359375 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 113.421875 disk.rbd2.out‚106.7744140625$ memory_percent‚96.63553237915039 disk.rbd1.in‚0( disk.nvme0c0n1.out‚18113.60693359375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 113.421875 disk.sda.in‚0 disk.sda.out‚ 130.640625* proc.memory.percent‚17.056703567504883 network.recv‚ 1.173705e+06 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd2.in‚0& proc.memory.availableMB‚ 551.234375 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 cpu‚25.322917324544694 disk.nvme0n1.in‚ 113.421875O6Ÿ52¼- _wandbruntime‚3246  _runtime‚3246*Ô[ɽà _runtime‚3246.668860367 _step‚349 ' lr-AdamW/pg1‚0.00018444118769870592 ' lr-AdamW/pg2‚0.00018444118769870592 trainer/global_step‚4299 % _timestamp‚1.7884315619640028e+09ΈåàÎ¾È trainer/global_step‚4299 % _timestamp‚1.7884315619640028e+09 _runtime‚3246.668860367 _step‚349 ' lr-AdamW/pg1‚0.00018444118769870592 ' lr-AdamW/pg2‚0.00018444118769870592Ý;A2¿- _wandbruntime‚3246  _runtime‚3246={ï9À×O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249295854975488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62318730 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3593588295168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3594244657664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246354997414446 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252889425267712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚3246.669735897 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62318730 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36357358 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20519501 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚246354325937330 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62318732 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272277340898816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242773982379008 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81040813 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36357356 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3593588295168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81020775 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3770229218304 L?DeviceStatsMonitor.on_train_batc"à#v¤ h_start/active.small_pool.freed‚81020775 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252889425267712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276047546177024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56877310 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249295836972544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276047737935360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _step‚350 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252890099633152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242773965339648 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143359545 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3770397036544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56876857 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249295854975488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62318732 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143339505 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3594244657664Å(üÉâ'ÁÜO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62318730 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3593588295168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3594244657664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246354997414446 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252889425267712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚3246.669735897 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62318730 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36357358 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20519501 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚246354325937330 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62318732 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272277340898816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242773982379008 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81040813 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36357356 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3593588295168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81020775 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3770229218304 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81020775 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252889425267712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276047546177024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56877310 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249295836972544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276047737935360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _step‚350 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252890099633152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242773965339648 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143359545 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3770397036544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56876857 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249295854975488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62318732 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143339505 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3594244657664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249295854975488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844Þ˜§÷n2Â- _wandbruntime‚3247  _runtime‚3247õ(ºç&ÃáM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3594407723008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276110241292800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚242828886509568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143392041 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249352190011392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143392041 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272338991636480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚351 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81059293 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249352208014336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252946597734400 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143371975 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242828869470208 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81059293 trainer/global_step‚4299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20524044 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3595064098816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3581831470610 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62332746 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3595064098816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚3247.343899747 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62332748 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252947272113152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246410046502854 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56889568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36365526 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3581177032646 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884315626385946e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36365524 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3771249656320ˆA‡Ç>%ÄæM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272338967696384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81039229 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249352190011392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62332748 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62332746 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81039229 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚246410717980178 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20524489 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276110047450624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81059293 trainer/global_step‚4299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20524044 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3595064098816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3581831470610 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62332746 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3595064098816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚3247.343899747 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62332748 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252947272113152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246410046502854 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56889568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36365526 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3581177032646 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884315626385946e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36365524 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3771249656320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3594407723008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276110241292800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚242828886509568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143392041 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249352190011392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143392041 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272338991636480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚351 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81059293 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249352208014336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252946597734400 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143371975 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242828869470208 E+)[ÆÕ 1loss/FBt1/assignment_loss‚0.10757613363396454 0loss/FBt3/assignment_loss‚0.3168583116044333 .loss/FBt4/detection_loss‚3.038321171488081 0loss/FRt1/assignment_loss‚1.3797936591696232 2loss/SRqqt1/assignment_loss‚1.2275920759838743 1loss/SRqqt3/detection_loss‚2.8524997028811225 /loss/FBt2/detection_loss‚2.9984885255495706 2loss/SRqqt3/assignment_loss‚1.9726931732790223 /loss/FBt1/detection_loss‚0.6816637024155614 % _timestamp‚1.7884315626404665e+09 1loss/FBt2/assignment_loss‚0.10226520212988059 1loss/SRqqt1/detection_loss‚0.8611992516387732 0loss/FRt2/assignment_loss‚2.6947630869017707 2loss/SRqqt4/assignment_loss‚1.6693297778256238 .loss/FRt4/detection_loss‚2.817870318889618 _runtime‚ 3247.34514312 .loss/FBt3/detection_loss‚2.502219324762171 trainer/global_step‚4299 _step‚352 /loss/FRt4/assignment_loss‚4.001614167409785 2loss/SRqqt2/assignment_loss‚1.5781239930759459 0loss/FBt4/assignment_loss‚0.1001643378819738 /loss/FRt1/detection_loss‚1.7141929754709944 /loss/FRt3/detection_loss‚3.0754251969712123 0loss/SRqqt4/detection_loss‚2.201701061427593 &loss/total_loss‚45.538266103876595 0loss/FRt3/assignment_loss‚3.5684065095015933 /loss/FRt2/detection_loss‚2.3185843765735625 0loss/SRqqt2/detection_loss‚1.756920067410843 epoch‚3_6 ‰`ÇÚ 2loss/SRqqt3/assignment_loss‚1.9726931732790223 /loss/FBt1/detection_loss‚0.6816637024155614 % _timestamp‚1.7884315626404665e+09 1loss/FBt2/assignment_loss‚0.10226520212988059 1loss/SRqqt1/detection_loss‚0.8611992516387732 0loss/FRt2/assignment_loss‚2.6947630869017707 2loss/SRqqt4/assignment_loss‚1.6693297778256238 .loss/FRt4/detection_loss‚2.817870318889618 _runtime‚ 3247.34514312 .loss/FBt3/detection_loss‚2.502219324762171 trainer/global_step‚4299 _step‚352 /loss/FRt4/assignment_loss‚4.001614167409785 2loss/SRqqt2/assignment_loss‚1.5781239930759459 0loss/FBt4/assignment_loss‚0.1001643378819738 /loss/FRt1/detection_loss‚1.7141929754709944 /loss/FRt3/detection_loss‚3.0754251969712123 0loss/SRqqt4/detection_loss‚2.201701061427593 &loss/total_loss‚45.538266103876595 0loss/FRt3/assignment_loss‚3.5684065095015933 /loss/FRt2/detection_loss‚2.3185843765735625 0loss/SRqqt2/detection_loss‚1.756920067410843 epoch‚3 1loss/FBt1/assignment_loss‚0.10757613363396454 0loss/FBt3/assignment_loss‚0.3168583116044333 .loss/FBt4/detection_loss‚3.038321171488081 0loss/FRt1/assignment_loss‚1.3797936591696232 2loss/SRqqt1/assignment_loss‚1.2275920759838743 1loss/SRqqt3/detection_loss‚2.8524997028811225 /loss/FBt2/detection_loss‚2.9984885255495706à”«N4“È‚(:ˆ Ò™åÔÝ´ÔŽ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.083' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êN`ä/É‚(:¤ Ò™åÔïãÐ disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 131.08203125 disk.sda.in‚0 disk.sda.out‚ 131.08203125' proc.memory.availableMB‚ 551.4609375" proc.memory.rssMB‚ 2794.5703125 disk.rbd1.out‚ 1.83984375 disk.dm-1.in‚0* proc.memory.percent‚17.056703567504883& disk.nvme0n1.out‚18114.07470703125 disk.rbd3.in‚0 disk.dm-0.in‚ 113.73046875 disk.sda1.in‚0# memory_percent‚96.6341495513916 disk.sda2.in‚0 disk.rbd2.out‚106.8525390625 disk.drbd1777.out‚0 disk.rbd3.out‚0 network.recv‚ 1.177291e+06 disk.drbd1777.in‚0 cpu‚25.321676798015382 disk.rbd0.in‚0 network.sent‚ 9.322822e+06) disk./.usagePercent‚73.87451842934749 disk.nvme0n1.in‚ 113.73046875 disk.rbd1.in‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 113.73046875 disk.sda2.out‚0# disk.dm-0.out‚18114.07470703125 disk.sda1.out‚0 disk.dm-1.out‚ 131.08203125% disk./.usageGB‚1100.7837104797363( disk.nvme0c0n1.out‚18114.07470703125%"S“Ê‚(:ˆ á™åÔúõí gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.5466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k}ís'Ë‚(:œ á™åÔþðµ" disk.dm-0.out‚18213.7568359375 disk.sda.in‚0% disk./.usageGB‚1100.7836532592773 disk.dm-1.out‚ 131.703125' disk.nvme0c0n1.out‚18213.7568359375 disk.sda2.in‚0 disk.nvme0n1.in‚ 113.75390625% disk.nvme0n1.out‚18213.7568359375 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375( proc.memory.availableMB‚ 551.40234375 disk.sda2.out‚0 disk.dm-0.in‚ 113.75390625 network.sent‚ 9.431745e+06) disk./.usagePercent‚73.87451458923448 disk.rbd0.in‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 113.75390625 disk.sda3.out‚ 131.703125 disk.rbd3.in‚0" proc.memory.rssMB‚ 2794.5703125* proc.memory.percent‚17.056703567504883 cpu‚25.320367887797733 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0$ memory_percent‚96.63450717926025 network.recv‚ 1.18354e+06 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd2.out‚108.4267578125 disk.sda.out‚ 131.703125?B|š2Ì- _wandbruntime‚3282  _runtime‚32823í+ÈÍ trainer/global_step‚4349 % _timestamp‚1.7884315976385515e+09 _runtime‚ 3282.34331298 _step‚353 ' lr-AdamW/pg1‚0.00018316048828251701 ' lr-AdamW/pg2‚0.00018316048828251701„{¡çÍÎÇ trainer/global_step‚4349 % _timestamp‚1.7884315976385515e+09 _runtime‚ 3282.34331298 _step‚353 ' lr-AdamW/pg1‚0.00018316048828251701 ' lr-AdamW/pg2‚0.00018316048828251701áK5åÐ2Ï- _wandbruntime‚3282  _runtime‚3282Ç&dÝ'Ð×O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3621195937090 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81966865 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4349 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255748737381376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _runtime‚3282.344261618 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚245519188907008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275359877782016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3813284329984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3635230463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63019530 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279172972450816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63019532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144986397 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144966357 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252113506917888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63019532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279173162112000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255748063015936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3635230463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81966865 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57511219 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3634574100992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36765758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36765756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81946827 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20745906 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255748737381376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20745463 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63019530 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255748063015936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3635230463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81966865 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57511219 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3634574100992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36765758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36765756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81946827 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20745906 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255748737381376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20745463 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63019530 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _runtime‚3282.344261618 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚245519188907008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275359877782016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3813284329984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3635230463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63019530 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279172972450816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63019532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144986397 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0âdv$2Ò- _wandbruntime‚3283  _runtime‚3283­aÀ ÓáM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4349 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36773926 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145018893 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20749906 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20750350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279235670302720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275421504579584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3636049904640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚245574075998208 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81985345 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81965281 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255805909861376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚355 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145018893 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144998827 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252169841953792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252169841953792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63033548 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57523830 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 38¼ß X13976075264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3635393528832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36773924 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249196759847586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255805909861376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3635393528832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275421528519680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚3283.011261652 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3622012372054 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81965281 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144998827ÆžöÌì&ÔæM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3622012372054 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81965281 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144998827 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4349 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36773926 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145018893 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20749906 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20750350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279235670302720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275421504579584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3636049904640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚245574075998208 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81985345 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81965281 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255805909861376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚355 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145018893 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144998827 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252169841953792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252169841953792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63033548 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57523830 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3813976075264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3635393528832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36773924 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249196759847586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255805909861376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3635393528832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275421528519680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚3283.011261652 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752ãŸ×>2Õ- _wandbruntime‚3283  _runtime‚3283]…ú]Ö× 2loss/SRqqt1/assignment_loss‚0.9879874287199386 /loss/FRt4/detection_loss‚2.2591078499952952 %loss/total_loss‚44.70720703738402 % _timestamp‚1.7884315983076978e+09 .loss/FRt1/detection_loss‚1.754949459841229 2loss/SRqqt2/assignment_loss‚1.3416421794614126 2loss/SRqqt4/assignment_loss‚1.3584879840879391 0loss/FBt1/assignment_loss‚0.1577761660158131 1loss/FBt2/assignment_loss‚0.19979416898762187 1loss/SRqqt3/detection_loss‚2.8379037057360015 .loss/FBt4/detection_loss‚2.148296291629473 epoch‚3 /loss/FRt3/assignment_loss‚4.042182987076895 /loss/FRt3/detection_loss‚2.8893092479024616 /loss/FBt2/detection_loss‚2.6542868773142496 trainer/global_step‚4349 _runtime‚3283.012609422 2loss/SRqqt3/assignment_loss‚1.9417613179733355 1loss/FBt3/assignment_loss‚0.24681892553538395 1loss/SRqqt1/detection_loss‚1.0471823092182757 1loss/FBt4/assignment_loss‚0.35112545887629193 /loss/FRt2/detection_loss‚2.3339208335830617 1loss/SRqqt2/detection_loss‚2.0838909360558486 1loss/SRqqt4/detection_loss‚2.0213755927979946 .loss/FBt1/detection_loss‚0.820003264400489 /loss/FBt3/detection_loss‚3.4082434243626065 /loss/FRt2/assignment_loss‚2.868153460896932 0loss/FRt4/assignment_loss‚3.3598127418094212 _step‚356 0loss/FRt1/assignment_loss‚1.5931944251060486Ú]ôb×Ü 2loss/SRqqt3/assignment_loss‚1.9417613179733355 1loss/FBt3/assignment_loss‚0.24681892553538395 1loss/SRqqt1/detection_loss‚1.0471823092182757 1loss/FBt4/assignment_loss‚0.35112545887629193 /loss/FRt2/detection_loss‚2.3339208335830617 1loss/SRqqt2/detection_loss‚2.0838909360558486 1loss/SRqqt4/detection_loss‚2.0213755927979946 .loss/FBt1/detection_loss‚0.820003264400489 /loss/FBt3/detection_loss‚3.4082434243626065 /loss/FRt2/assignment_loss‚2.868153460896932 0loss/FRt4/assignment_loss‚3.3598127418094212 _step‚356 0loss/FRt1/assignment_loss‚1.5931944251060486 2loss/SRqqt1/assignment_loss‚0.9879874287199386 /loss/FRt4/detection_loss‚2.2591078499952952 %loss/total_loss‚44.70720703738402 % _timestamp‚1.7884315983076978e+09 .loss/FRt1/detection_loss‚1.754949459841229 2loss/SRqqt2/assignment_loss‚1.3416421794614126 2loss/SRqqt4/assignment_loss‚1.3584879840879391 0loss/FBt1/assignment_loss‚0.1577761660158131 1loss/FBt2/assignment_loss‚0.19979416898762187 1loss/SRqqt3/detection_loss‚2.8379037057360015 .loss/FBt4/detection_loss‚2.148296291629473 epoch‚3 /loss/FRt3/assignment_loss‚4.042182987076895 /loss/FRt3/detection_loss‚2.8893092479024616 /loss/FBt2/detection_loss‚2.6542868773142496 trainer/global_step‚4349 _runtime‚3283.012609422äË«½Y“Ø‚(:ˆ ð™åÔÞðá gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚105.929' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1858 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é~'Ù‚(:œ ð™åÔæý« disk.sda.out‚ 132.26953125 network.recv‚ 1.187777e+06 disk.dm-0.in‚ 113.7578125" disk.dm-0.out‚18312.9560546875 disk.dm-1.in‚0 disk.dm-1.out‚ 132.26953125" proc.memory.rssMB‚ 2794.5703125) disk./.usagePercent‚73.87452278147558 disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 551.421875* proc.memory.percent‚17.056703567504883 disk.sda2.in‚0 network.sent‚ 9.43721e+06% disk./.usageGB‚1100.7837753295898 disk.sda3.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚18312.9560546875 disk.nvme0n1.in‚ 113.7578125 disk.rbd1.in‚0' disk.nvme0c0n1.out‚18312.9560546875 disk.sda2.out‚0 cpu‚25.319144177969847! disk.nvme0c0n1.in‚ 113.7578125 disk.drbd1777.in‚0 disk.rbd2.out‚108.5048828125 proc.cpu.threads‚17 disk.sda.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 132.26953125 disk.rbd1.out‚ 1.83984375 disk.rbd0.in‚0# memory_percent‚96.6343879699707ªÂ3žÚ‚(:“ ÿ™åÔÁ‘í gpu.0.gpu‚50.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.517599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã1½•)Û‚(:ž ÿ™åԾ膑 disk.dm-0.in‚ 113.7578125# disk.dm-0.out‚18416.03173828125 disk.sda.in‚0 disk.dm-1.out‚ 132.8046875 disk.nvme0n1.in‚ 113.7578125) disk./.usagePercent‚73.87473501172146 disk.rbd2.in‚0( proc.memory.availableMB‚ 550.69921875" proc.memory.rssMB‚ 2794.5703125 cpu‚25.31799447594491 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 9.546077e+06 disk.sda1.out‚0$ memory_percent‚96.63879871368408* proc.memory.percent‚17.056703567504883 disk.sda3.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 132.8046875 disk.dm-1.in‚0 disk.rbd2.out‚108.5048828125 network.recv‚ 1.194059e+06$ disk./.usageGB‚1100.786937713623! disk.nvme0c0n1.in‚ 113.7578125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 1.83984375( disk.nvme0c0n1.out‚18416.03173828125 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda.out‚ 132.8046875& disk.nvme0n1.out‚18416.03173828125û©"s“Ü‚(:ˆ ŽšåÔ˜êß gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.197' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3856222551552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3675559906816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚254931158860288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82872879 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82872879 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146593209 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258606700764160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3676216269312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63720332 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚258607375129600 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278442390725120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚258607375129600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚258606700764160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3675559906816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248264378395648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63720330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248264395435008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3676216269312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58146609 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _runtime‚3318.218107281 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37174158 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63720332 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_stŠ¶e–&art/segment.large_pool.peak‚430 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3856222551552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3675559906816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚254931158860288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82872879 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82872879 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146593209 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258606700764160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3676216269312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63720332 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚258607375129600 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278442390725120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚258607375129600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚258606700764160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3675559906816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248264378395648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63720330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248264395435008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3676216269312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58146609 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _runtime‚3318.218107281 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37174158 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63720332 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚254987493896192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚258663873230848 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58159303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37182326 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146645745 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚278504065402880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3676379334656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251982130237670 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82911397 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚254987511899136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63734348 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚254987493896192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3857081773056 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3677035710464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3663502149426 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146625679 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3676379334656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58159740 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚359 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0@sÏfë&æåM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146625679 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3676379334656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58159740 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚359 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82891333 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251982801714994 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚254987493896192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚258663873230848 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58159303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37182326 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146645745 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚278504065402880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3676379334656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251982130237670 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82911397 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚254987511899136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63734348 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚254987493896192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3857081773056 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3677035710464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3663502149426ç˜ÌQ2ç- _wandbruntime‚3318  _runtime‚3318ôwû·\èÖ 1loss/FBt3/assignment_loss‚0.32402985317942995 -loss/FRt1/detection_loss‚1.63535670096224 1loss/SRqqt2/detection_loss‚1.9544375153879325 epoch‚3 % _timestamp‚1.7884316341806176e+09 /loss/FRt2/detection_loss‚2.3105716005615564 1loss/SRqqt4/detection_loss‚1.8748226791035896 /loss/FBt3/detection_loss‚3.1733442743619285 .loss/FBt4/detection_loss‚2.914855365242277 %loss/total_loss‚45.62397314305432 0loss/FRt1/assignment_loss‚1.2075027877634221 0loss/FRt2/assignment_loss‚3.3330105530179064 2loss/SRqqt3/assignment_loss‚2.2339516644542283 0loss/FBt2/assignment_loss‚0.3278237535851076 1loss/FBt4/assignment_loss‚0.46048572207135813 /loss/FBt1/detection_loss‚0.7542528273297648 /loss/FBt2/detection_loss‚3.1225365102291107 /loss/FRt4/assignment_loss‚3.115384739764193 2loss/SRqqt1/assignment_loss‚0.8222106854963516 /loss/FRt3/detection_loss‚3.0778659274501186 1loss/SRqqt1/detection_loss‚1.0888700375067337 2loss/SRqqt2/assignment_loss‚1.5320934030848246 _runtime‚3318.885255235 /loss/FRt3/assignment_loss‚4.239822333858859 1loss/FBt1/assignment_loss‚0.18324151292633967 .loss/FRt4/detection_loss‚2.206924868390915 1loss/SRqqt3/detection_loss‚2.4256371799353005 _step‚360 trainer/global_step‚4399 2loss/SRqqt4/assignment_loss‚1.3049406473908354aÉÕaéÛ 1loss/SRqqt2/detection_loss‚1.9544375153879325 epoch‚3 % _timestamp‚1.7884316341806176e+09 /loss/FRt2/detection_loss‚2.3105716005615564 1loss/SRqqt4/detection_loss‚1.8748226791035896 /loss/FBt3/detection_loss‚3.1733442743619285 .loss/FBt4/detection_loss‚2.914855365242277 %loss/total_loss‚45.62397314305432 0loss/FRt1/assignment_loss‚1.2075027877634221 0loss/FRt2/assignment_loss‚3.3330105530179064 2loss/SRqqt3/assignment_loss‚2.2339516644542283 0loss/FBt2/assignment_loss‚0.3278237535851076 1loss/FBt4/assignment_loss‚0.46048572207135813 /loss/FBt1/detection_loss‚0.7542528273297648 /loss/FBt2/detection_loss‚3.1225365102291107 /loss/FRt4/assignment_loss‚3.115384739764193 2loss/SRqqt1/assignment_loss‚0.8222106854963516 /loss/FRt3/detection_loss‚3.0778659274501186 1loss/SRqqt1/detection_loss‚1.0888700375067337 2loss/SRqqt2/assignment_loss‚1.5320934030848246 _runtime‚3318.885255235 /loss/FRt3/assignment_loss‚4.239822333858859 1loss/FBt1/assignment_loss‚0.18324151292633967 .loss/FRt4/detection_loss‚2.206924868390915 1loss/SRqqt3/detection_loss‚2.4256371799353005 _step‚360 trainer/global_step‚4399 2loss/SRqqt4/assignment_loss‚1.3049406473908354 1loss/FBt3/assignment_loss‚0.32402985317942995 -loss/FRt1/detection_loss‚1.63535670096224è®ã$ @ê‚(:“ šåÔÿù†Ž gpu.0.gpu‚57.0 gpu.0.memory‚¹“×C^8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.905' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.381000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00¦W ë‚(:• šåÔ„ÀÑ% disk./.usageGB‚1100.7870063781738% disk.nvme0c0n1.out‚18515.79296875 disk.sda2.in‚0" proc.memory.rssMB‚ 2794.5703125* proc.memory.percent‚17.056703567504883 network.sent‚ 9.557555e+06 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 550.6640625 cpu‚25.31553073577642! disk.rbd2.out‚108.60302734375! disk.nvme0c0n1.in‚ 113.7578125 disk.rbd0.out‚0 disk.sda.out‚ 133.8046875# disk.nvme0n1.out‚18515.79296875 disk.rbd3.in‚0 disk.sda.in‚0 network.recv‚ 1.20179e+06) disk./.usagePercent‚73.87473961985708 disk.sda2.out‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0$ memory_percent‚96.63901329040527 proc.cpu.threads‚17 disk.sda3.out‚ 133.8046875 disk.sda1.in‚0 disk.nvme0n1.in‚ 113.7578125 disk.dm-0.in‚ 113.7578125 disk.dm-0.out‚18515.79296875 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.sda1.out‚0 disk.dm-1.out‚ 133.8046875͆Pœì‚(:‘ ¬šåÔò⌎ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.7' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.339999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åÕÅoí‚(:“ ¬šåÔî² network.sent‚ 9.665525e+06 disk.drbd1777.out‚0# proc.memory.availableMB‚550.875) disk./.usagePercent‚73.87474602004544 disk.rbd0.in‚0! disk.rbd2.out‚110.19287109375 disk.sda3.out‚ 134.25390625 disk.dm-1.out‚ 134.25390625 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 113.7578125 disk.dm-0.out‚ 18614.8671875 disk.sda1.in‚0" disk.nvme0n1.out‚ 18614.8671875 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.sda.out‚ 134.25390625$ memory_percent‚96.63772583007812* proc.memory.percent‚17.056703567504883 disk.nvme0n1.in‚ 113.7578125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.in‚0 network.recv‚ 1.207567e+06 disk.dm-1.in‚0 disk.rbd1.out‚ 1.83984375! disk.nvme0c0n1.in‚ 113.7578125 disk.rbd2.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1100.7871017456055 disk.sda1.out‚0 cpu‚25.314352947395566$ disk.nvme0c0n1.out‚ 18614.8671875" proc.memory.rssMB‚ 2794.5703125ÚQF¬2î- _wandbruntime‚3353  _runtime‚3353¾v&Éïà ' lr-AdamW/pg1‚0.00018055569004570113 ' lr-AdamW/pg2‚0.00018055569004570113 trainer/global_step‚4449 % _timestamp‚1.7884316688435562e+09 _runtime‚3353.548283061 _step‚3614ßÝ‘ÎðÈ ' lr-AdamW/pg1‚0.00018055569004570113 ' lr-AdamW/pg2‚0.00018055569004570113 trainer/global_step‚4449 % _timestamp‚1.7884316688435562e+09 _runtime‚3353.548283061 _step‚361éB!t2ñ- _wandbruntime‚3353  _runtime‚3353`ÍYôÝ'ò×O ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3702866615906 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281524951548416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚261466012877824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚257748792799744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37582558 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281524927608320 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261465338512384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3716545712640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83798931 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21200054 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21199627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148240101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64421132 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83818969 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257748810802688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚254712451539554 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37582556 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3899262718464 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58782612 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3717202075136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58782183 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 6544377566†Tâ'óÜO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281524951548416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚261466012877824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚257748792799744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37582558 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281524927608320 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261465338512384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3716545712640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83798931 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21200054 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21199627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148240101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64421132 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83818969 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257748810802688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚254712451539554 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37582556 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3899262718464 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58782612 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3717202075136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58782183 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3702866615906êî‹ÿ/2ô- _wandbruntime‚3354  _runtime‚3354©njæˆ"õàM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261523185357824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64435146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83837449 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148272597 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261523185357824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83817385 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3900122072064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3717365140480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257805163841536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3717365140480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64435148 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3899960558592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257805145838592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64435146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148272597 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281586602286080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254768172105078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884316695020678e+09 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3718021516288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freer4ŽB^d‚58794844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64435148 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83837449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148252531 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281586578345984 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚363£¯ë&öåM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254768172105078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884316695020678e+09 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3718021516288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58794844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64435148 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83837449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148252531 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281586578345984 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚363 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261523185357824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64435146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83837449 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148272597 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261523185357824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83817385 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3900122072064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3717365140480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257805163841536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3717365140480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64435148 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3899960558592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257805145838592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64435146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148272597 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281586602286080ëšüfe2÷- _wandbruntime‚3354  _runtime‚3354Ñ=ÍìWøÑ /loss/FRt2/assignment_loss‚3.620928873886933 /loss/FBt3/assignment_loss‚0.554931548377499 /loss/FRt1/detection_loss‚1.5404504375918857 /loss/FBt1/detection_loss‚0.6354985025817852 /loss/FBt2/detection_loss‚3.2667137601158838 % _timestamp‚1.7884316695042195e+09 2loss/SRqqt2/assignment_loss‚1.7816974446177483 2loss/SRqqt3/assignment_loss‚1.7585979162192926 1loss/FBt4/assignment_loss‚0.13115654729263043 1loss/SRqqt1/detection_loss‚1.1234850813634694 _step‚364 0loss/FRt1/assignment_loss‚1.5131629997829221 1loss/SRqqt4/detection_loss‚2.0286627469993217 .loss/FRt4/detection_loss‚2.104330441781453 1loss/SRqqt2/detection_loss‚1.9935671778825612 0loss/SRqqt3/detection_loss‚2.292199983829405 .loss/FBt4/detection_loss‚2.995982676744461 epoch‚3 trainer/global_step‚4449 _runtime‚3354.208892273 1loss/FBt1/assignment_loss‚0.24683048585607456 1loss/FBt2/assignment_loss‚0.09646917325021191 .loss/FRt2/detection_loss‚2.731488552447912 0loss/FRt3/assignment_loss‚3.1196258001857333 2loss/SRqqt1/assignment_loss‚0.9227795927436091 -loss/FBt3/detection_loss‚2.81272177696228 &loss/total_loss‚45.391051176872416 0loss/SRqqt4/assignment_loss‚1.79862988031492 0loss/FRt4/assignment_loss‚3.4836394786834717 -loss/FRt3/detection_loss‚2.83750029736095®ñ)“\ùÖ /loss/FRt2/assignment_loss‚3.620928873886933 /loss/FBt3/assignment_loss‚0.554931548377499 /loss/FRt1/detection_loss‚1.5404504375918857 /loss/FBt1/detection_loss‚0.6354985025817852 /loss/FBt2/detection_loss‚3.2667137601158838 % _timestamp‚1.7884316695042195e+09 2loss/SRqqt2/assignment_loss‚1.7816974446177483 2loss/SRqqt3/assignment_loss‚1.7585979162192926 1loss/FBt4/assignment_loss‚0.13115654729263043 1loss/SRqqt1/detection_loss‚1.1234850813634694 _step‚364 0loss/FRt1/assignment_loss‚1.5131629997829221 1loss/SRqqt4/detection_loss‚2.0286627469993217 .loss/FRt4/detection_loss‚2.104330441781453 1loss/SRqqt2/detection_loss‚1.9935671778825612 0loss/SRqqt3/detection_loss‚2.292199983829405 .loss/FBt4/detection_loss‚2.995982676744461 epoch‚3 trainer/global_step‚4449 _runtime‚3354.208892273 1loss/FBt1/assignment_loss‚0.24683048585607456 1loss/FBt2/assignment_loss‚0.09646917325021191 .loss/FRt2/detection_loss‚2.731488552447912 0loss/FRt3/assignment_loss‚3.1196258001857333 2loss/SRqqt1/assignment_loss‚0.9227795927436091 -loss/FBt3/detection_loss‚2.81272177696228 &loss/total_loss‚45.391051176872416 0loss/SRqqt4/assignment_loss‚1.79862988031492 0loss/FRt4/assignment_loss‚3.4836394786834717 -loss/FRt3/detection_loss‚2.83750029736095ì±R_ò’ú‚(:‡ »šåÔÂÛ¡Ž gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚97.827' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.5654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|¾6R#û‚(:˜ »šåÔ¡Ÿ‘ network.recv‚ 1.21242e+06 disk.nvme0n1.in‚ 113.7578125 disk.rbd2.in‚0* proc.memory.percent‚17.056703567504883 disk.rbd3.in‚0 disk.dm-0.in‚ 113.7578125$ disk.nvme0n1.out‚18714.662109375! disk.nvme0c0n1.in‚ 113.7578125& disk.nvme0c0n1.out‚18714.662109375 disk.sda2.out‚0 disk.rbd0.in‚0! disk.rbd2.out‚110.19287109375 disk.sda.out‚ 134.85546875 disk.rbd0.out‚0 cpu‚25.313168780320645# disk./.usageGB‚1100.78755569458 disk.sda3.in‚0! disk.dm-0.out‚18714.662109375 disk.sda1.in‚0 disk.rbd1.in‚0$ memory_percent‚96.64063453674316 disk.sda1.out‚0 disk.rbd1.out‚ 1.83984375 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda2.in‚0 network.sent‚ 9.671806e+06 disk.sda3.out‚ 134.85546875 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 134.85546875' proc.memory.availableMB‚ 550.3984375" proc.memory.rssMB‚ 2794.5703125' disk./.usagePercent‚73.874776484942 disk.drbd1777.out‚0 disk.sda.in‚0…k%²ü‚(:’ ÊšåÔ¸ãý gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.59' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.317999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁO]ý‚(:’ ÊšåÔ ãˆ& disk.nvme0c0n1.out‚18813.708984375 disk.sda2.out‚0( proc.memory.availableMB‚ 550.62890625 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.056703567504883) disk./.usagePercent‚73.87477904501735 disk.dm-0.in‚113.8125 disk.dm-1.out‚ 135.3515625! disk.rbd2.out‚110.20849609375 disk.sda3.in‚0 disk.sda3.out‚ 135.3515625" proc.memory.rssMB‚ 2794.5703125 disk.drbd1777.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 network.recv‚ 1.218625e+06 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.rbd0.out‚0 network.sent‚ 9.780053e+06! disk.dm-0.out‚18813.708984375 disk.sda.out‚ 135.3515625 disk.nvme0n1.in‚113.8125 disk.rbd0.in‚0 cpu‚25.312099699612897% disk./.usageGB‚1100.7875938415527 disk.sda.in‚0 disk.nvme0c0n1.in‚113.8125$ memory_percent‚96.63922786712646$ disk.nvme0n1.out‚18813.708984375™4Fž2þ- _wandbruntime‚3389  _runtime‚3389¼yªlÈÿ % _timestamp‚1.7884317045286653e+09 _runtime‚ 3389.23346031 _step‚365 ' lr-AdamW/pg1‚0.00017923213406112248 ' lr-AdamW/pg2‚0.00017923213406112248 trainer/global_step‚4499[´¦Í€Ç ' lr-AdamW/pg1‚0.00017923213406112248 ' lr-AdamW/pg2‚0.00017923213406112248 trainer/global_step‚4499 % _timestamp‚1.7884317045286653e+09 _runtime‚ 3389.23346031 _step‚365íé´Ä2- _wandbruntime‚3389  _runtime‚3389¡4#¼Ý'‚×O YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚264324650626048 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149846913 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260566444742144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59416624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚264323976260608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3942063425536 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚253754791451648 % _timestamp‚1.7884317045294316e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚257498493406962 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264323976260608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3758187880960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65121932 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288549527917056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚3389.234409118 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65121930 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚257499164884078 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260566462745088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260566462745088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253754808491008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3743701955314 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84745021 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚264324650626048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84724983 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260566462745088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253754808491008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3743701955314 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84745021 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚264324650626048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84724983 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264323976260608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3758187880960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65121932 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288549527917056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚3389.234409118 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65121930 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚257499164884078 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260566462745088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149899449 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚253809712621568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84763501 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257554885449810 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149879383 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚367 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65135946 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59428804 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4499 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84743437 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149879383 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3759007322112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21429680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288612226225664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944›=Çðì&†æM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65135948 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚367 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65135946 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59428804 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4499 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84743437 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149879383 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3759007322112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21429680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288612226225664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149899449 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚253809712621568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84763501 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257554885449810 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149879383 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3799173686784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3799173686784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65822730 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3785191732478 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151473765 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65822730 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267182614008832 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65822732 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3985087132160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267182614008832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263384114687488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85671073 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85651035 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85671073 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260284535274370 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3784537294722 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263384096684544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38399356 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291674931174400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 % _timestamp‚1.7884317397473438e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263384096684544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287690001374720Ú83þâ'•ÜO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21650898 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3984929799680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3424.452319962 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291675112446976 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60049810 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151493805 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263384114687488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3798517324288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85651035 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38399356 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291674931174400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 % _timestamp‚1.7884317397473438e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263384096684544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287690001374720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256500015019008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65822732 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3798517324288 _step‚370 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3799173686784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3799173686784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65822730 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3785191732478 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151473765 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65822730 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267182614008832 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65822732 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3985087132160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267182614008832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263384114687488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85671073 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85651035 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85671073 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260284535274370 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3784537294722 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263384096684544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151526301 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287751676052480 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85669489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38407526 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3799336752128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚3425.337084966 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65836746 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267240460854272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85689553 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21654892 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3786008167650 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65836746 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85669489 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256554902110208 % _timestamp‚1.7884317406316075e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263440449723392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60062416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65836748 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38407524 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚260340255839894 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263440467726336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3799993127936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256554919149568 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260340927317218 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291737620271104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚371 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60062865 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3985772219392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚4549 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151526301 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263440467726336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263440449723392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151506235 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952nç隘æM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65836748 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38407524 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚260340255839894 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263440467726336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3799993127936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256554919149568 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260340927317218 P@DeviceStatsMonitor.on_train_batch_end/reserved_bÇ Å]ytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291737620271104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚371 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60062865 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3985772219392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚4549 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151526301 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263440467726336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263440449723392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151506235 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151526301 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287751676052480 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85669489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38407526 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3799336752128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚3425.337084966 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65836746 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267240460854272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85689553 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21654892 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3786008167650 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65836746 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85669489 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256554902110208 % _timestamp‚1.7884317406316075e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263440449723392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60062416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2ó¼S¦2™- _wandbruntime‚3425  _runtime‚3425d^ônYšÓ _runtime‚3425.338162647 /loss/FRt4/assignment_loss‚3.344656086393765 2loss/SRqqt2/assignment_loss‚1.3967409545653744 /loss/FRt4/detection_loss‚2.2047835497983863 % _timestamp‚1.7884317406334784e+09 1loss/FBt1/assignment_loss‚0.17497814166742798 .loss/FRt2/detection_loss‚2.360092195107582 0loss/SRqqt2/detection_loss‚2.032508747880919 2loss/SRqqt1/assignment_loss‚0.9654379544703111 /loss/FBt2/assignment_loss‚0.590645293200699 1loss/FBt4/assignment_loss‚0.41645199013873935 epoch‚3 _step‚372 /loss/FRt1/assignment_loss‚1.232003859111241 0loss/FRt2/assignment_loss‚2.8137079566083054 .loss/FRt1/detection_loss‚1.898319805154995 /loss/FBt3/detection_loss‚2.6629001877524634 /loss/FRt3/assignment_loss‚3.479062393307686 2loss/SRqqt3/assignment_loss‚1.3683472697223935 0loss/FBt3/assignment_loss‚0.3668259982226697 1loss/SRqqt1/detection_loss‚0.8936410140603372 /loss/FBt2/detection_loss‚2.6634755730628967 /loss/FBt4/detection_loss‚3.2675732459340776 2loss/SRqqt4/assignment_loss‚1.5629697971873813 1loss/SRqqt3/detection_loss‚2.3093439153262545 $loss/total_loss‚43.8910990690847 /loss/FRt3/detection_loss‚2.9133422262966633 1loss/SRqqt4/detection_loss‚1.8361654513412051 /loss/FBt1/detection_loss‚1.1371254627729335 trainer/global_step‚4549…§>^›Ø 2loss/SRqqt1/assignment_loss‚0.9654379544703111 /loss/FBt2/assignment_loss‚0.590645293200699 1loss/FBt4/assignment_loss‚0.41645199013873935 epoch‚3 _step‚372 /loss/FRt1/assignment_loss‚1.232003859111241 0loss/FRt2/assignment_loss‚2.8137079566083054 .loss/FRt1/detection_loss‚1.898319805154995 /loss/FBt3/detection_loss‚2.6629001877524634 /loss/FRt3/assignment_loss‚3.479062393307686 2loss/SRqqt3/assignment_loss‚1.3683472697223935 0loss/FBt3/assignment_loss‚0.3668259982226697 1loss/SRqqt1/detection_loss‚0.8936410140603372 /loss/FBt2/detection_loss‚2.6634755730628967 /loss/FBt4/detection_loss‚3.2675732459340776 2loss/SRqqt4/assignment_loss‚1.5629697971873813 1loss/SRqqt3/detection_loss‚2.3093439153262545 $loss/total_loss‚43.8910990690847 /loss/FRt3/detection_loss‚2.9133422262966633 1loss/SRqqt4/detection_loss‚1.8361654513412051 /loss/FBt1/detection_loss‚1.1371254627729335 trainer/global_step‚4549 _runtime‚3425.338162647 /loss/FRt4/assignment_loss‚3.344656086393765 2loss/SRqqt2/assignment_loss‚1.3967409545653744 /loss/FRt4/detection_loss‚2.2047835497983863 % _timestamp‚1.7884317406334784e+09 1loss/FBt1/assignment_loss‚0.17497814166742798 .loss/FRt2/detection_loss‚2.360092195107582 0loss/SRqqt2/detection_loss‚2.032508747880919ô÷hV`žœ‚(:“ †›åÔ—Ìê gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚126.612' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.322400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y:0)‚(:ž †›åÔÝÄþ% disk.nvme0n1.out‚19112.1474609375 disk.rbd1.out‚ 1.83984375 disk.sda2.out‚0" proc.memory.rssMB‚ 2794.5703125 network.recv‚ 1.236876e+06) disk./.usagePercent‚73.87478877330365 disk.sda.out‚ 137.7109375 disk.nvme0n1.in‚ 117.96484375# memory_percent‚96.6374158859253( proc.memory.availableMB‚ 550.92578125% disk./.usageGB‚1100.7877388000488 disk.sda3.out‚ 137.7109375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0" disk.dm-0.out‚19112.1474609375 disk.dm-1.in‚0 disk.dm-1.out‚ 137.7109375 disk.sda.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 9.905888e+06' disk.nvme0c0n1.out‚19112.1474609375 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.056703567504883 disk.dm-0.in‚ 117.96484375 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 117.96484375 disk.rbd0.out‚0 disk.rbd2.out‚ 111.962890625 cpu‚25.307412630438428 disk.rbd3.in‚0 disk.sda1.in‚02´â@ž‚(:’ •›åÔˆêŽ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚92.144' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.428800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯Õ=,Ÿ‚(:¡ •›åÔŸ† disk.dm-0.in‚ 117.96484375 disk.sda.out‚ 138.3203125 disk.sda3.in‚0* proc.memory.percent‚17.056703567504883 proc.cpu.threads‚17 disk.dm-1.out‚ 138.3203125 disk.sda2.out‚0$ memory_percent‚96.63739204406738' proc.memory.availableMB‚ 550.9296875 network.sent‚ 1.0014817e+07 disk.rbd3.in‚0& disk.nvme0n1.out‚19211.72314453125" disk.nvme0c0n1.in‚ 117.96484375 disk.rbd0.out‚0 disk.drbd1777.in‚0" proc.memory.rssMB‚ 2794.5703125 network.recv‚ 1.243558e+06 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 1.83984375( disk.nvme0c0n1.out‚19211.72314453125 disk.rbd0.in‚0 disk.rbd2.out‚ 112.087890625$ disk./.usageGB‚1100.787841796875 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 cpu‚25.306299839859868# disk.dm-0.out‚19211.72314453125 disk.nvme0n1.in‚ 117.96484375 disk.sda3.out‚ 138.3203125) disk./.usagePercent‚73.87479568550708êy(×2 - _wandbruntime‚3460  _runtime‚3460²—ÂèÉ¡Ã ' lr-AdamW/pg1‚0.00017654408743755882 ' lr-AdamW/pg2‚0.00017654408743755882 trainer/global_step‚4599 % _timestamp‚1.7884317755780535e+09 _runtime‚3460.282784601 _step‚373€ÚãÎ¢È ' lr-AdamW/pg1‚0.00017654408743755882 ' lr-AdamW/pg2‚0.00017654408743755882 trainer/global_step‚4599 % _timestamp‚1.7884317755780535e+09 _runtime‚3460.282784601 _step‚373õž±2£- _wandbruntime‚3460  _runtime‚3460°.$Ý'¤×O ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3839503130112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21876051 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153100617 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86577087 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚266201766629888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚294800603271168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263071248618894 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3840159492608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259245221547008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270041926122496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚294800417804288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270041251757056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3840159492608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3839503130112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66523532 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290772562198016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4028041073152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259245204507648 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66523530 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86577087 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3826027071886 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4599 _step‚374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38807756 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360È1qð ¥ÜO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4027879546368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86597125 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38807758 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66523530 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153100617 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86577087 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚266201766629888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚294800603271168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263071248618894 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3840159492608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259245221547008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270041926122496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚294800417804288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270041251757056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3840159492608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3839503130112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66523532 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290772562198016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4028041073152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259245204507648 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66523530 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86577087 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3826027071886 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4599 _step‚374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38807756 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3839503130112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21876051 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚Ñ5©öò0 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259300125677568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884317762389004e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60696503 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚266258101665792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚270099098602496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263126297707302 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66537546 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38815926 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263126969184626 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266258119668736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3840322557952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66537548 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚266258119668736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3840978933760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60696945 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3826843507058 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3826189069094 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3840978933760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290834188995584 trainer/global_step‚4599 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86615605 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚294863114573312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21880579 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4028901637632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153133087 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153153153 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66537548 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153133087 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚294862927022592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3840322557952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66537546 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266258101665792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38815924 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚3460.944447341 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚270098424223744 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4028738027008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259300108638208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430M=‡«ì&¨æM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3840978933760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290834188995584 trainer/global_step‚4599 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86615605 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚294863114573312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21880579 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4028901637632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153133087 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153153153 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66537548 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153133087 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚294862927022592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3840322557952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66537546 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266258101665792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38815924 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚3460.944447341 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚270098424223744 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4028738027008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259300108638208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259300125677568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884317762389004e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60696503 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚266258101665792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚270099098602496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263126297707302 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66537546 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38815926 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263126969184626 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266258119668736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3840322557952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66537548 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚266258119668736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3840978933760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60696945 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3826843507058 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3826189069094 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944÷ç;—ù2©- _wandbruntime‚3460  _runtime‚3460³`•GZªÔ 1loss/SRqqt2/detection_loss‚1.7016013110021375 1loss/SRqqt4/assignment_loss‚1.590791505606224 1loss/FBt1/assignment_loss‚0.14084383999060826 /loss/FRt2/detection_loss‚2.2046243469670133 /loss/FRt3/detection_loss‚2.8924673240760277 1loss/SRqqt1/detection_loss‚0.9672467920767225 _step‚376 /loss/FRt3/assignment_loss‚4.510539256293198 .loss/FRt1/detection_loss‚1.944425608497113 /loss/FBt2/detection_loss‚2.4957012712955473 &loss/total_loss‚44.679864000687566 trainer/global_step‚4599 0loss/FBt4/assignment_loss‚0.2596104640646705 /loss/FRt2/assignment_loss‚2.837689013975971 1loss/SRqqt4/detection_loss‚1.8890581093728542 .loss/FBt3/detection_loss‚2.317659556865692 .loss/FBt4/detection_loss‚3.189340216772897 2loss/SRqqt1/assignment_loss‚1.1123738809070294 2loss/SRqqt3/assignment_loss‚2.1129328288576184 0loss/FBt2/assignment_loss‚0.6051920258090832 /loss/FRt4/detection_loss‚2.0877772094087397 _runtime‚3460.945841739 /loss/FBt1/detection_loss‚0.6729169053492544 epoch‚3 0loss/FRt1/assignment_loss‚1.4083161515494187 0loss/FRt4/assignment_loss‚3.6953372410003174 2loss/SRqqt2/assignment_loss‚1.5808718370379142 0loss/FBt3/assignment_loss‚0.2678775742650032 1loss/SRqqt3/detection_loss‚2.1946697296465145 $ _timestamp‚1.788431776240827e+0913_«Ù 0loss/FBt2/assignment_loss‚0.6051920258090832 /loss/FRt4/detection_loss‚2.0877772094087397 _runtime‚3460.945841739 /loss/FBt1/detection_loss‚0.6729169053492544 epoch‚3 0loss/FRt1/assignment_loss‚1.4083161515494187 0loss/FRt4/assignment_loss‚3.6953372410003174 2loss/SRqqt2/assignment_loss‚1.5808718370379142 0loss/FBt3/assignment_loss‚0.2678775742650032 1loss/SRqqt3/detection_loss‚2.1946697296465145 $ _timestamp‚1.788431776240827e+09 1loss/SRqqt2/detection_loss‚1.7016013110021375 1loss/SRqqt4/assignment_loss‚1.590791505606224 1loss/FBt1/assignment_loss‚0.14084383999060826 /loss/FRt2/detection_loss‚2.2046243469670133 /loss/FRt3/detection_loss‚2.8924673240760277 1loss/SRqqt1/detection_loss‚0.9672467920767225 _step‚376 /loss/FRt3/assignment_loss‚4.510539256293198 .loss/FRt1/detection_loss‚1.944425608497113 /loss/FBt2/detection_loss‚2.4957012712955473 &loss/total_loss‚44.679864000687566 trainer/global_step‚4599 0loss/FBt4/assignment_loss‚0.2596104640646705 /loss/FRt2/assignment_loss‚2.837689013975971 1loss/SRqqt4/detection_loss‚1.8890581093728542 .loss/FBt3/detection_loss‚2.317659556865692 .loss/FBt4/detection_loss‚3.189340216772897 2loss/SRqqt1/assignment_loss‚1.1123738809070294 2loss/SRqqt3/assignment_loss‚2.1129328288576184øc­Küž¬‚(:“ ¤›åÔÙ¾Ô gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.391' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.478199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢W&e/­‚(:¤ ¤›åÔÿ£› disk.dm-1.out‚ 138.75390625' proc.memory.availableMB‚ 551.1640625$ disk./.usageGB‚1100.787841796875 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0) disk./.usagePercent‚73.87479568550708" disk.nvme0c0n1.in‚ 117.96484375( disk.nvme0c0n1.out‚19212.08154296875 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 1.0020221e+07 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 112.103515625 disk.sda3.in‚0 disk.sda3.out‚ 138.75390625# disk.dm-0.out‚19212.08154296875 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 117.96484375 disk.rbd1.out‚ 1.83984375 disk.rbd3.out‚0* proc.memory.percent‚17.056703567504883 disk.sda.in‚0 disk.nvme0n1.in‚ 117.96484375 disk.rbd1.in‚0 disk.sda2.in‚0$ memory_percent‚96.63596153259277 cpu‚25.305197150936134 network.recv‚ 1.247349e+06& disk.nvme0n1.out‚19212.08154296875 disk.rbd3.in‚0" proc.memory.rssMB‚ 2794.5703125 disk.sda.out‚ 138.75390625 disk.sda1.out‚0ç#€¢“®‚(:ˆ ³›åÔáûÐ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.087' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6174 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑW·8¯‚(:” ³›åÔõÄÏ disk.nvme0n1.in‚ 117.9765625 disk.rbd1.in‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 551.1484375" proc.memory.rssMB‚ 2794.5703125 network.recv‚ 1.253884e+06 disk.sda3.out‚ 139.359375* proc.memory.percent‚17.056703567504883 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.dm-0.out‚19311.37890625 disk.sda.out‚ 139.359375 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.rbd2.out‚112.18994140625 disk.sda.in‚0! disk.nvme0c0n1.in‚ 117.9765625% disk.nvme0c0n1.out‚19311.37890625 disk.sda2.in‚0 network.sent‚ 1.0129397e+07 disk.dm-0.in‚ 117.9765625 disk.dm-1.out‚ 139.359375# disk.nvme0n1.out‚19311.37890625 disk.rbd1.out‚ 1.83984375 disk.rbd0.out‚0) disk./.usagePercent‚73.87479568550708$ disk./.usageGB‚1100.787841796875 disk.sda2.out‚0$ memory_percent‚96.63605690002441 cpu‚25.304099191439494 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda1.in‚0¿¨¢“°‚(:ˆ ›åÔºŸß gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{~ M#±‚(:˜ ›åÔ€À³ disk.sda1.in‚0 disk.nvme0n1.in‚ 139.9296875# disk.nvme0n1.out‚19410.49609375 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0! disk.nvme0c0n1.in‚ 139.9296875 disk.rbd3.out‚0 disk.dm-0.in‚ 139.9296875$ memory_percent‚96.63617610931396% disk.nvme0c0n1.out‚19410.49609375 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 139.8203125 disk.drbd1777.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 network.recv‚ 1.257651e+06% disk./.usageGB‚1100.7878875732422 disk.rbd1.out‚ 1.83984375 disk.dm-0.out‚19410.49609375 disk.sda.in‚0" proc.memory.rssMB‚ 2794.5703125* proc.memory.percent‚17.056703567504883 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 139.8203125! disk.rbd2.out‚112.20556640625 disk.sda3.in‚0) disk./.usagePercent‚73.87479875759749 disk.dm-1.out‚ 139.8203125 network.sent‚ 1.013454e+07 disk.dm-1.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 551.12890625 cpu‚25.303020064978202ñV›,2²- _wandbruntime‚3496  _runtime‚3496saáȳ ' lr-AdamW/pg1‚0.00017518015698346116 ' lr-AdamW/pg2‚0.00017518015698346116 trainer/global_step‚4649 $ _timestamp‚1.788431811374065e+09 _runtime‚3496.078778078 _step‚377EˆÚ¤Í´Ç _step‚377 ' lr-AdamW/pg1‚0.00017518015698346116 ' lr-AdamW/pg2‚0.00017518015698346116 trainer/global_step‚4649 $ _timestamp‚1.788431811374065e+09 _runtime‚3496.078778078ù —ñ62µ- _wandbruntime‚3496  _runtime‚3496L-nr ¶×O VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22102878 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265856619009186 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67224332 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3881145298432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3880488935936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154747509 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67224330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61319454 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297925962301440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272899889505280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884318113747973e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67224330 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61319034 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269019418572288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297925776834560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚4649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved× Ãçk_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154727469 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261990411035648 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚3496.079763205 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87523177 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3881145298432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269019400569344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚293855099081216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293855075141120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22103296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265857290486302 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87503139 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272900563870720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67224332 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3880488935936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87503139 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚420 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚261990428075008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚269019418572288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39216158 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3866207973538 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39216156 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4070701693440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3866862411294 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272899889505280 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0b7â'·ÜO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39216156 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4070701693440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3866862411294 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272899889505280 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22102878 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265856619009186 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67224332 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3881145298432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3880488935936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154747509 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67224330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61319454 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297925962301440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272899889505280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884318113747973e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67224330 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61319034 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269019418572288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297925776834560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚4649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154727469 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261990411035648 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚3496.079763205 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87523177 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3881145298432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269019400569344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚293855099081216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293855075141120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22103296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265857290486302 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87503139 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272900563870720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67224332 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3880488935936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87503139 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚420 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚261990428075008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚269019418572288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39216158 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3866207973538úˆc½H2¸- _wandbruntime‚3496  _runtime‚34963Ñ›¥å&¹ßM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297988475746304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22107842 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293916725878784 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61331732 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87541657 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293916749818880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265912339574710 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3867678846466 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚4649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154759939 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 # _timestamp‚1.78843181202746e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154780005 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87521593 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3881964739584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154759939 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67238346 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269075753608192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚265913011052034 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚3496.732696159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87521593 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚272957736350720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39224326 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22107408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚269075771611136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3881308363776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297988292389888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262045332205568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3867024408502 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚379 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87541657 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3881964739584 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154780005 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67238346$Î9DyºäM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297988292389888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262045332205568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3867024408502 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚379 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87541657 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3881964739584 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154780005 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67238346 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297988475746304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22107842 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293916725878784 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_spl˜A´üqit.all.freed‚61331732 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87541657 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293916749818880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265912339574710 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3867678846466 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚4649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154759939 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 # _timestamp‚1.78843181202746e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154780005 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87521593 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3881964739584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154759939 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67238346 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269075753608192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚265913011052034 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚3496.732696159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87521593 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚272957736350720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39224326 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22107408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚269075771611136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3881308363776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844ûü$2»- _wandbruntime‚3496  _runtime‚3496KLDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3921474741760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _step‚382 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88449229 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88429191 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275758527253504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3531.533606171 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3907697750702 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271837070514688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67925130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22329910 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268643332353710 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3922131104256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4113864276992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271837052511744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271837070514688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156354321 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156374361 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3907043312946 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264735617563648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39624556 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3921474741760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67925132 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275759201618944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296937612024320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3922131104256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88429191 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67925132 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301051500241408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884318468286982e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275758527253504 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275759201618944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67925130 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268642660876594 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88449229 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39624558 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301051312677376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61954468 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296937635964416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4113700653056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4699 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61954018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22329462æú­ â'ÇÜO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156354321 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156374361 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3907043312946 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264735617563648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39624556 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3921474741760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67925132 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275759201618944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296937612024320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3922131104256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88429191 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67925132 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301051500241408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884318468286982e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275758527253504 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275759201618944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67925130 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268642660876594 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88449229 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39624558 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301051312677376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61954468 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296937635964416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4113700653056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4699 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61954018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22329462 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264735634603008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3921474741760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _step‚382 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88449229 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88429191 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275758527253504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3531.533606171 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3907697750702 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271837070514688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67925130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22329910 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268643332353710 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3922131104256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4113864276992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271837052511744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271837070514688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0þO»ÓQ2È- _wandbruntime‚3532  _runtime‚3532DUMÚÉàM EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271893405550592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚268698381442118 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296999262761984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61966649 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264790521694208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_end/inactÀI›H !ive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3922950545408 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67939146 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 $ _timestamp‚1.788431847497062e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296999286702080 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚383 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚301114013855232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264790538733568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22334363 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22333925 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚4699 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88467709 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3908514185874 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275816374098944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268699052919442 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275815699720192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88467709 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271893423553536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156386791 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4114727153152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39632724 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3922294169600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156406857 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚301113828401664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88447645 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271893423553536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67939146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88447645 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3922294169600 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67939148 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61966649 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264790521694208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3922950545408 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67939146 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 $ _timestamp‚1.788431847497062e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296999286702080 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚383 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚301114013855232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264790538733568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22334363 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22333925 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚4699 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88467709 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3908514185874 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275816374098944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268699052919442 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275815699720192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88467709 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271893423553536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156386791 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4114727153152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39632724 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3922294169600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156406857 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚301113828401664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88447645 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271893423553536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67939146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88447645 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3922294169600 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67939148 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271893405550592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚268698381442118 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296999262761984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ÿ;ÌJ2Ë- _wandbruntime‚3532  _runtime‚3532}TòôZÌÔ .loss/FRt3/detection_loss‚2.899891730510827 0loss/FBt3/assignment_loss‚0.6439142673172884 /loss/FRt1/detection_loss‚1.6214641213180527 1loss/SRqqt3/detection_loss‚2.3967632406797166 0loss/FRt1/assignment_loss‚0.9334245636349633 /loss/FRt2/assignment_loss‚3.079507429152727 /loss/FRt4/assignment_loss‚3.987985067566236 1loss/FBt2/assignment_loss‚0.48683871515095234 /loss/FBt1/detection_loss‚0.5962999224527786 2loss/SRqqt2/assignment_loss‚1.8594090990266021 1loss/SRqqt1/detection_loss‚1.0276887446928482 2loss/SRqqt1/assignment_loss‚0.8622407228088914 2loss/SRqqt4/assignment_loss‚1.5782211393606467 1loss/FBt1/assignment_loss‚0.15444997808401917 .loss/FRt4/detection_loss‚2.172370813290278 _runtime‚3532.203863221 epoch‚3 trainer/global_step‚4699 % _timestamp‚1.7884318474991856e+09 /loss/FRt3/assignment_loss‚2.928258256478743 2loss/SRqqt3/assignment_loss‚2.2325086815234942 0loss/FBt4/assignment_loss‚0.2617779725114815 1loss/SRqqt2/detection_loss‚2.0354920765575097 1loss/SRqqt4/detection_loss‚1.7329784617072246 /loss/FBt2/detection_loss‚3.4400283247232437 .loss/FBt3/detection_loss‚2.142905010117425 .loss/FBt4/detection_loss‚2.289878476411104 %loss/total_loss‚43.93756603949228 _step‚384 /loss/FRt2/detection_loss‚2.5732692244152227$þ¸_ÍÙ 0loss/FBt3/assignment_loss‚0.6439142673172884 /loss/FRt1/detection_loss‚1.6214641213180527 1loss/SRqqt3/detection_loss‚2.3967632406797166 0loss/FRt1/assignment_loss‚0.9334245636349633 /loss/FRt2/assignment_loss‚3.079507429152727 /loss/FRt4/assignment_loss‚3.987985067566236 1loss/FBt2/assignment_loss‚0.48683871515095234 /loss/FBt1/detection_loss‚0.5962999224527786 2loss/SRqqt2/assignment_loss‚1.8594090990266021 1loss/SRqqt1/detection_loss‚1.0276887446928482 2loss/SRqqt1/assignment_loss‚0.8622407228088914 2loss/SRqqt4/assignment_loss‚1.5782211393606467 1loss/FBt1/assignment_loss‚0.15444997808401917 .loss/FRt4/detection_loss‚2.172370813290278 _runtime‚3532.203863221 epoch‚3 trainer/global_step‚4699 % _timestamp‚1.7884318474991856e+09 /loss/FRt3/assignment_loss‚2.928258256478743 2loss/SRqqt3/assignment_loss‚2.2325086815234942 0loss/FBt4/assignment_loss‚0.2617779725114815 1loss/SRqqt2/detection_loss‚2.0354920765575097 1loss/SRqqt4/detection_loss‚1.7329784617072246 /loss/FBt2/detection_loss‚3.4400283247232437 .loss/FBt3/detection_loss‚2.142905010117425 .loss/FBt4/detection_loss‚2.289878476411104 %loss/total_loss‚43.93756603949228 _step‚384 /loss/FRt2/detection_loss‚2.5732692244152227 .loss/FRt3/detection_loss‚2.899891730510827€âKü“΂(:ˆ ï›åÔ¡€°Ž gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.087' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6174 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W—¼i&Ï‚(:› ï›åÔ’É–‘! disk.rbd2.out‚113.92041015625 disk.rbd3.in‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 141.33203125# memory_percent‚96.6362714767456 disk.rbd1.out‚ 1.83984375 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 139.9296875" proc.memory.rssMB‚ 2794.5703125 disk.sda.out‚ 141.33203125( proc.memory.availableMB‚ 551.11328125 disk.sda3.out‚ 141.33203125% disk./.usageGB‚1100.7872428894043 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 139.9296875 disk.sda.in‚0! disk.nvme0c0n1.in‚ 139.9296875 network.recv‚ 1.272907e+06 disk.rbd3.out‚0! disk.dm-0.out‚19609.365234375$ disk.nvme0n1.out‚19609.365234375& disk.nvme0c0n1.out‚19609.365234375* proc.memory.percent‚17.056703567504883 cpu‚25.2997456436185 proc.cpu.threads‚17 network.sent‚ 1.0255081e+07 disk.sda2.in‚0 disk.rbd1.in‚0( disk./.usagePercent‚73.8747554923242ìý$žÐ‚(:“ þ›åÔðôÓ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.734' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.746800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@záˆ,Ñ‚(:¡ þ›åÔ„Öà( disk./.usagePercent‚73.8747554923242 disk.rbd0.out‚0 disk.rbd3.out‚0# disk.dm-0.out‚19708.52685546875 disk.dm-1.out‚ 141.8203125( disk.nvme0c0n1.out‚19708.52685546875* proc.memory.percent‚17.056703567504883 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda1.in‚0 network.sent‚ 1.0362854e+07 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1777.in‚0! disk.nvme0c0n1.in‚ 139.9296875$ memory_percent‚96.63918018341064" proc.memory.rssMB‚ 2794.5703125 network.recv‚ 1.278516e+06 disk.rbd0.in‚0! disk.rbd2.out‚113.93994140625 disk.nvme0n1.in‚ 139.9296875 disk.sda2.in‚0 disk.dm-0.in‚ 139.9296875 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375( proc.memory.availableMB‚ 550.63671875 cpu‚25.298720322750345% disk./.usageGB‚1100.7872428894043 disk.rbd2.in‚0 disk.sda3.out‚ 141.8203125 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda.out‚ 141.8203125& disk.nvme0n1.out‚19708.52685546875 disk.sda2.out‚0»þ2Ò- _wandbruntime‚3567  _runtime‚3567õú[ÉÓà ' lr-AdamW/pg1‚0.00017241390293671065 ' lr-AdamW/pg2‚0.00017241390293671065 trainer/global_step‚4749 % _timestamp‚1.7884318825692437e+09 _runtime‚3567.274024437 _step‚385 ÞyòÎÔÈ % _timestamp‚1.7884318825692437e+09 _runtime‚3567.274024437 _step‚385 ' lr-AdamW/pg1‚0.00017241390293671065 ' lr-AdamW/pg2‚0.00017241390293671065 trainer/global_step‚4749@Ä”2Õ- _wandbruntime‚3567  _runtime‚3567N¿nD>ÖÖO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278617165001728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304176980673536 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304176795206656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271428702744002 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22556241 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300020172847616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89375281 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4156646299136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962460547584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3948533090110 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68625932 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278617839367168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40032956 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3963116910080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68625930 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274654722457088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274654704454144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚KEË£ž 185466880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚4749 _step‚386 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158001213 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267480824091648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157981173 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68625932 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278617165001728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304176980673536 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304176795206656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271428702744002 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22556241 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300020172847616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89375281 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4156646299136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962460547584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3948533090110 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68625932 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278617839367168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40032956 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3963116910080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68625930 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274654722457088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274654704454144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚4749 _step‚386 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158001213 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267480824091648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157981173 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68625932 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158033709 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304239290335232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158013643 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89373697 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274711075495936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 # _timestamp‚1.78843188323971e+09 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267535745261568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89393761 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22560700 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68639946 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948695087318 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62601824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274711057492992 trainer/global_step‚4749 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300081823585280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3949349525282 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68639948 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4157666883584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158013643 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚271484423309526 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025%fê&ÚäM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89373697 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267535728222208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40041124 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68639948 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278675011847168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _step‚387 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚278674337468416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3963279975424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚278675011847168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158033709 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304239290335232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158013643 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89373697 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274711075495936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 # _timestamp‚1.78843188323971e+09 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267535745261568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89393761 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22560700 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68639946 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948695087318 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62601824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274711057492992 trainer/global_step‚4749 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300081823585280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3949349525282 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68639948 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4157666883584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158013643 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚271484423309526 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048ƒ¶GA(2Û- _wandbruntime‚3567  _runtime‚3567áÜÃËÜÖ /loss/FRt3/detection_loss‚2.9376181853669032 1loss/SRqqt3/detection_loss‚2.0422603165781177 /loss/FBt2/detection_loss‚2.7641819053226047 $loss/total_loss‚46.1138936184353 trainer/global_step‚4749 _runtime‚3567.946436348 _step‚388 1loss/FBt2/assiïC‹e?gnment_loss‚0.20830720427653027 1loss/FBt4/assignment_loss‚0.05804571376315185 0loss/FRt4/assignment_loss‚3.6309043740232787 2loss/SRqqt2/assignment_loss‚1.9563012972828888 1loss/FBt1/assignment_loss‚0.23706067013436324 /loss/FBt1/detection_loss‚0.6367671486147399 1loss/SRqqt4/detection_loss‚1.9019363291086036 % _timestamp‚1.7884318832415297e+09 0loss/FRt2/assignment_loss‚3.5607009654821353 2loss/SRqqt4/assignment_loss‚1.9166687245064593 0loss/FBt3/assignment_loss‚0.2880722996778786 /loss/FRt2/detection_loss‚2.4691904866418173 /loss/FRt4/detection_loss‚2.3731858916580677 epoch‚3 0loss/FRt1/assignment_loss‚1.3023009796937306 /loss/FRt3/assignment_loss‚3.662257488284792 2loss/SRqqt1/assignment_loss‚0.8635810706459108 -loss/FRt1/detection_loss‚1.84551183351626 0loss/SRqqt1/detection_loss‚1.037604315298014 0loss/SRqqt2/detection_loss‚2.072432437113353 /loss/FBt3/detection_loss‚2.6199905342525907 /loss/FBt4/detection_loss‚3.8008800404412404 2loss/SRqqt3/assignment_loss‚1.9281334067518647PèÜaÝÛ 0loss/FRt2/assignment_loss‚3.5607009654821353 2loss/SRqqt4/assignment_loss‚1.9166687245064593 0loss/FBt3/assignment_loss‚0.2880722996778786 /loss/FRt2/detection_loss‚2.4691904866418173 /loss/FRt4/detection_loss‚2.3731858916580677 epoch‚3 0loss/FRt1/assignment_loss‚1.3023009796937306 /loss/FRt3/assignment_loss‚3.662257488284792 2loss/SRqqt1/assignment_loss‚0.8635810706459108 -loss/FRt1/detection_loss‚1.84551183351626 0loss/SRqqt1/detection_loss‚1.037604315298014 0loss/SRqqt2/detection_loss‚2.072432437113353 /loss/FBt3/detection_loss‚2.6199905342525907 /loss/FBt4/detection_loss‚3.8008800404412404 2loss/SRqqt3/assignment_loss‚1.9281334067518647 /loss/FRt3/detection_loss‚2.9376181853669032 1loss/SRqqt3/detection_loss‚2.0422603165781177 /loss/FBt2/detection_loss‚2.7641819053226047 $loss/total_loss‚46.1138936184353 trainer/global_step‚4749 _runtime‚3567.946436348 _step‚388 1loss/FBt2/assignment_loss‚0.20830720427653027 1loss/FBt4/assignment_loss‚0.05804571376315185 0loss/FRt4/assignment_loss‚3.6309043740232787 2loss/SRqqt2/assignment_loss‚1.9563012972828888 1loss/FBt1/assignment_loss‚0.23706067013436324 /loss/FBt1/detection_loss‚0.6367671486147399 1loss/SRqqt4/detection_loss‚1.9019363291086036 % _timestamp‚1.7884318832415297e+09„×3Æ“Þ‚(:ˆ œåÔ›ê© gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.734' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7468 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øÑ@ñ+ß‚(:  œåÔÆÃé" proc.memory.rssMB‚ 2794.5703125) disk./.usagePercent‚73.87475958844475 disk.rbd3.out‚0 disk.dm-0.in‚ 139.9296875* proc.memory.percent‚17.056703567504883 disk.sda.in‚0 disk.dm-1.out‚ 142.57421875 disk.sda3.in‚0 disk.rbd3.in‚0# memory_percent‚96.6390609741211 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1777.out‚0# disk.dm-0.out‚19808.00537109375 proc.cpu.threads‚17 disk.sda.out‚ 142.57421875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 139.9296875 disk.rbd1.in‚0 disk.drbd1777.in‚0 network.sent‚ 1.0368241e+07 disk.rbd1.out‚ 1.83984375 disk.sda2.in‚0& disk.nvme0n1.out‚19808.00537109375( disk.nvme0c0n1.out‚19808.00537109375 disk.rbd2.out‚114.0263671875 disk.sda3.out‚ 142.57421875% disk./.usageGB‚1100.7873039245605 disk.rbd0.out‚0 disk.rbd2.in‚0% proc.memory.availableMB‚ 550.65625 cpu‚25.297667130776365 network.recv‚ 1.282268e+06! disk.nvme0c0n1.in‚ 139.9296875 disk.rbd0.in‚0ž}¤žà‚(:“ œœåÔÕ놎 gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.606' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.121199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›‚ü„&á‚(:› œœåÔŽ£Ù& disk.nvme0n1.out‚19907.16552734375( disk.nvme0c0n1.out‚19907.16552734375 network.recv‚ 1.295573e+06 disk.dm-0.in‚ 139.9296875 disk.sda1.out‚0 disk.rbd2.out‚114.0419921875$ memory_percent‚96.63822650909424 disk.sda3.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0) disk./.usagePercent‚73.87476342855776% disk./.usageGB‚1100.7873611450195 disk.sda.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 139.9296875! disk.nvme0c0n1.in‚ 139.9296875 disk.rbd3.out‚0# disk.dm-0.out‚19907.16552734375 cpu‚25.296640138263136 disk.rbd1.out‚ 1.83984375 disk.sda2.in‚0 disk.sda.out‚ 142.90625 disk.sda3.out‚ 142.90625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 142.90625" proc.memory.rssMB‚ 2794.5703125 network.sent‚ 1.0480276e+07 disk.rbd3.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.056703567504883 disk.sda1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0( proc.memory.availableMB‚ 550.792968759af‹žâ‚(:“ «œåÔ²”æ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚147.391' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.478199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02€Û\%ã‚(:š «œåÔ©–´‘ disk.sda2.in‚0 disk.sda3.out‚ 143.3828125( proc.memory.availableMB‚ 550.69140625 cpu‚25.295540808036165% disk./.usageGB‚1100.7874908447266 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 139.9296875 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0# memory_percent‚96.6388463973999 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 143.3828125 network.recv‚ 1.300202e+06 disk.sda1.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17$ disk.nvme0n1.out‚19907.615234375" proc.memory.rssMB‚ 2794.5703125 network.sent‚ 1.0486556e+07 disk.sda.out‚ 143.3828125 disk.sda2.out‚0 disk.rbd2.in‚0* proc.memory.percent‚17.056703567504883 disk.drbd1777.out‚0& disk.nvme0c0n1.out‚19907.615234375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 139.9296875 disk.dm-0.in‚ 139.9296875! disk.dm-0.out‚19907.615234375 disk.rbd2.out‚115.6318359375 disk.rbd1.in‚0) disk./.usagePercent‚73.87477213281393 disk.sda.in‚0 disk.rbd1.out‚ 1.83984375\Ä!2ä- _wandbruntime‚3602  _runtime‚3602£ÍïüÇåÁ & lr-AdamW/pg1‚0.0001710121558272549 & lr-AdamW/pg2‚0.0001710121558272549 trainer/global_step‚4799 % _timestamp‚1.7884319176818597e+09 _runtime‚3602.386599076 _step‚389©UÌæÆ _runtime‚3602.386599076 _step‚389 & lr-AdamW/pg1‚0.0001710121558272549 & lr-AdamW/pg2‚0.0001710121558272549 trainer/global_step‚4799 % _timestamp‚1.7884319176818597e+09…(k]k2ç- _wandbruntime‚3602  _runtime‚3602*±ºÍÝ'è×O PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90301333 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90301333 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281475802749952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307302311872000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274215416088526 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281476477115392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90281295 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚390 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63225299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40441356 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚277472356396544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303102685790720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69326732 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3988713991762 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚277472356396544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159628065 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274214744611410 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69326732 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22784375 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚3602.387535481 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4004102715904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281475802749952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159608025 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚277472374399488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69326730 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884319176826026e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303102709730816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90281295 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4004102715904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚270226030619648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4003446353408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚277472374399488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4003446353408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0}ôh[â'éÜO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281476477115392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90281295 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚390 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63225299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40441356 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚277472356396544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303102685790720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69326732 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3988713991762 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚277472356396544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159628065 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274214744611410 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69326732 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22784375 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚3602.387535481 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4004102715904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281475802749952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159608025 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚277472374399488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69326730 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884319176826026e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303102709730816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90281295 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4004102715904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚270226030619648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4003446353408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚277472374399488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4003446353408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90301333 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90301333 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281475802749952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307302311872000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274215416088526 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844†z”q2ê- _wandbruntime‚3603  _runtime‚3603¦pIÕÃëáM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22788421 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚391 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274271136654258 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚270280934750208 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚277528727438336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚277528727438336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90299749 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4004922157056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚277528709435392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4004265781248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159660561 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4200654050816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281533649595392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159640495 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2Ÿw&$605 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3989530426726 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281533649595392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90319813 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884319183448403e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22788854 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990184864690 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69340746 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40449524 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307365014519296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4200488343040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63237945 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303164336528384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚3603.050186314 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63238380 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69340746 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159660561 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4004922157056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307364824871424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90319813 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚277528727438336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚277528727438336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90299749 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4004922157056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚277528709435392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4004265781248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159660561 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4200654050816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281533649595392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159640495 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3989530426726 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281533649595392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90319813 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884319183448403e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22788854 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990184864690 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69340746 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40449524 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307365014519296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4200488343040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63237945 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303164336528384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚3603.050186314 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63238380 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69340746 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159660561 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4004922157056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307364824871424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90319813 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚281362090245120 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚281362108248064 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 161701844 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 % _timestamp‚1.7884319908455253e+09 _step‚393 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 161701844 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚285413932557824 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚285414443390976 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚91343005 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4249740881408 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4051842312704 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚23175444 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚307239697966080 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚41062268 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚152 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚278024311438430 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚281362108248064 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 161717823 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128µ`B:H*üÂT [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4249740881408 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4051842312704 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚23175444 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚307239697966080 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚41062268 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚152 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚278024311438430 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚281362108248064 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 161717823 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4036925819398 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚281362090245120 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚281362108248064 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 161701844 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 % _timestamp‚1.7884319908455253e+09 _step‚393 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 161701844 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚285413932557824 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚285414443390976 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚91343005 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376‰ó½âø2ý- _wandbruntime‚3676  _runtime‚36766*y%N)þÈR SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚70385622 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚41068772 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚285451638546432 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚311527679707648 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4250106008064 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚281398920273920 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884319917353165e+09 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚64248764 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚70385624 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 161735931 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚307277573699584 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115343872 _runtime‚3676.440389082 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚91334330 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚278059968807326 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚285451127713280 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 161719952>Yvy|ÿÍR \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 161735931 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚307277573699584 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115343872 _runtime‚3676.440389082 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚91334330 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚278059968807326 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚285451127713280 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 161719952 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚70385622 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚41068772 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚285451638546432 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚311527679707648 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4250106008064 BDH× _batch_end/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚91350307 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚274022188464128 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 4052700269568 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚281398938276864 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4052207439360 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚281398920273920 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884319917353165e+09 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚64248764 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚70385624 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521Š Boá2€- _wandbruntime‚3689  _runtime‚3689µÖ™ât›§ ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.021278882399201393 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023362306877970695 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.0957864448428154 particle/accuracy_3_of_5‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_2_of_5‚0.38903605937957764 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3601851463317871 particle/accuracy_5_of_6‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17411589622497559 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4597829580307007 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9822197556495667 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9975579977035522 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9817798137664795 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17131978273391724 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_1_of_1‚0.46954938769340515 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4342135190963745 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9783058166503906 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8989346623420715 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.24436935782432556 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9947378039360046 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3752436339855194 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 particle/accuracy_2_of_7‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18808165192604065 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.667571485042572 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4623866379261017 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10540593415498734 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 particle/accuracy_3_of_7‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 trainer/global_step‚4835 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.372333824634552 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4579962193965912 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007558579556643963 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1834334284067154 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16872799396514893 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24444444477558136 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22762346267700195 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19255492091178894 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.010392469353973866 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.33368054032325745 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2664141356945038 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38966232538223267 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3735087811946869 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3295454680919647 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9884404540061951 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23750747740268707 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 epoch‚3 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34343037009239197 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9897739291191101 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9866546988487244 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4638778567314148 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.031007753685116768 *jet/accuracy_1_of_3‚0.8472012281417847 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_1_of_3‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 _runtime‚3689.200535012 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.20820340514183044 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38576069474220276 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23059602081775665 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121212482452393 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.19555555284023285 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 &particle/f_score‚0.327353298664093 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4524192214012146 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 jet/accuracy_6_of_6‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5739824175834656 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_5‚0.012384792789816856 particle/accuracy_1_of_4‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02527581714093685 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23427796363830566 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5014375448226929 particle/accuracy_6_of_7‚1 *loss/val_total_loss‚44.861258082506026 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2536824643611908 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9698275327682495 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13274934887886047 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10187532007694244 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 particle/accuracy_5_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 % _timestamp‚1.7884320044936604e+09 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2812865376472473 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.38856586813926697 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.35773807764053345 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_6‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2428189516067505 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2505227327346802 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8153820633888245 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.014779875054955482 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9900810718536377 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_7‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.45201045274734497 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929478168487549 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36148327589035034 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3262094557285309 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.486607164144516 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.991460919380188 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2565132975578308 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33686599135398865 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.47893449664115906 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.47270113229751587 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2128615528345108 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 +particle/specificity‚0.9428170323371887 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17888353765010834 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6341003775596619 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23473870754241943 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3583333492279053 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3841802775859833 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5216999650001526 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05288645252585411 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.47529393434524536 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.434408038854599 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4583333432674408 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3374335467815399 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17219610512256622 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9325398206710815 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06025518849492073 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15107598900794983 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4552469253540039 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9753968119621277 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 jet/accuracy_7_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4649123251438141 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 particle/accuracy_6_of_6‚1 jet/accuracy_2_of_7‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.528002917766571 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07088718563318253 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7361757755279541 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3870370388031006 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3872606158256531 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03297491371631622 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7validation_average_jet_accuracy‚0.41978970170021057 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4930555522441864 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28101274371147156 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2028011977672577 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18772466480731964 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5364011526107788 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3805628716945648 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48544561862945557 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.42982470989227295 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4217773973941803 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20689651370048523 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003846153849735856 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18235105276107788 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.24366195499897003 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19118645787239075 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 +jet/accuracy_2_of_3‚0.21573889255523682 ,jet/accuracy_4_of_4‚0.004405052866786718 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4860233664512634 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.03947368264198303 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1350574791431427 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 +jet/accuracy_4_of_6‚0.01923076994717121 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_6‚1 particle/accuracy_7_of_7‚0.5 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3392255902290344 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967495203018188 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07446808367967606 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9600694179534912 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5768178105354309 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973205924034119 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1784611940383911 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961740970611572 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5593422055244446 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5089285373687744 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4553571343421936 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3303303122520447 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9725422263145447 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4645901024341583 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9449564218521118 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9847952723503113 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07824164628982544 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05119047686457634 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2369121015071869 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21459786593914032 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12448559701442719 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21905478835105896 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5252147912979126 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4185529947280884 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3565877079963684 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4385965168476105 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.2563016414642334 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2167578488588333 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5486452579498291 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08376705646514893 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11100342124700546 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2120632529258728 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013235841877758503 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43869107961654663 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05962643772363663 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9868838787078857 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4217284917831421 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.20577600598335266 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 jet/accuracy_5_of_6‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43800318241119385 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9850649833679199 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9857777953147888 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4801599383354187 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02950832061469555 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25474613904953003 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4118509888648987 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_puri:¸Å½Àty‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2916020452976227 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07522246986627579 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4281609356403351 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4337919056415558 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010355253703892231 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.521812379360199 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5511854887008667 jet/accuracy_5_of_7‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.43097352981567383 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39498910307884216 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.26591548323631287 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08740007877349854 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.43778344988822937 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0173611119389534 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22521944344043732 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022741014137864113 particle/accuracy_2_of_2‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.08974359929561615 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4904961884021759 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2103249877691269 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009542534127831459 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3482142984867096 particle/accuracy_4_of_4‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3541666865348816 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5267360806465149 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9922136068344116 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2807539403438568 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16649426519870758 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4638778567314148 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3580247163772583 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 _step‚395 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2905172109603882 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +jet/accuracy_3_of_3‚0.01627030223608017 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_1_of_7‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45600420236587524 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9834979772567749 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4544544219970703 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4499461054801941 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.095238097012043 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16872799396514893 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18114754557609558 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_1_of_5‚1 jet/accuracy_1_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4451117515563965 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9874655604362488 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37733912467956543 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.578696072101593 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.48279279470443726 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.446422278881073 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19933390617370605 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8356326222419739 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5196883678436279 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14240966737270355 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5034722089767456 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 particle/accuracy_4_of_7‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39048007130622864 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5133333206176758 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9884737730026245 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.37179484963417053 ("Purity/2FRt2SRqqt2FBt/event_purity‚04,+2a‚ § <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9817798137664795 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17131978273391724 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_1_of_1‚0.46954938769340515 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4342135190963745 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9783058166503906 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8989346623420715 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.24436935782432556 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9947378039360046 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3752436339855194 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 particle/accuracy_2_of_7‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18808165192604065 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.667571485042572 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4623866379261017 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10540593415498734 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 particle/accuracy_3_of_7‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 trainer/global_step‚4835 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.372333824634552 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4579962193965912 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007558579556643963 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1834334284067154 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16872799396514893 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24444444477558136 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22762346267700195 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19255492091178894 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.010392469353973866 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.33368054032325745 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2664141356945038 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38966232538223267 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3735087811946869 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3295454680919647 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9884404540061951 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23750747740268707 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 epoch‚3 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34343037009239197 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9897739291191101 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9866546988487244 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4638778567314148 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.031007753685116768 *jet/accuracy_1_of_3‚0.8472012281417847 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_1_of_3‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 _runtime‚3689.200535012 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.20820340514183044 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38576069474220276 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23059602081775665 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121212482452393 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.19555555284023285 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 &particle/f_score‚0.327353298664093 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4524192214012146 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 jet/accuracy_6_of_6‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5739824175834656 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_5‚0.012384792789816856 particle/accuracy_1_of_4‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02527581714093685 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23427796363830566 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5014375448226929 particle/accuracy_6_of_7‚1 *loss/val_total_loss‚44.861258082506026 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2536824643611908 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9698275327682495 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13274934887886047 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10187532007694244 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 particle/accuracy_5_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 % _timestamp‚1.7884320044936604e+09 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2812865376472473 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.38856586813926697 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.35773807764053345 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_6‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2428189516067505 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2505227327346802 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8153820633888245 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.014779875054955482 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9900810718536377 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_7‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.45201045274734497 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929478168487549 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36148327589035034 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3262094557285309 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.486607164144516 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.991460919380188 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2565132975578308 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33686599135398865 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.47893449664115906 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.47270113229751587 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2128615528345108 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 +particle/specificity‚0.9428170323371887 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17888353765010834 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6341003775596619 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23473870754241943 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3583333492279053 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3841802775859833 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5216999650001526 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05288645252585411 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.47529393434524536 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.434408038854599 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4583333432674408 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3374335467815399 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17219610512256622 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9325398206710815 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06025518849492073 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15107598900794983 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4552469253540039 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9753968119621277 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 jet/accuracy_7_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4649123251438141 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 particle/accuracy_6_of_6‚1 jet/accuracy_2_of_7‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.528002917766571 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07088718563318253 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7361757755279541 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3870370388031006 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3872606158256531 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03297491371631622 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7validation_average_jet_accuracy‚0.41978970170021057 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4930555522441864 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28101274371147156 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2028011977672577 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18772466480731964 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5364011526107788 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3805628716945648 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48544561862945557 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.42982470989227295 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4217773973941803 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20689651370048523 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003846153849735856 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18235105276107788 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.24366195499897003 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19118645787239075 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 +jet/accuracy_2_of_3‚0.21573889255523682 ,jet/accuracy_4_of_4‚0.004405052866786718 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4860233664512634 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.03947368264198303 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1350574791431427 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 +jet/accuracy_4_of_6‚0.01923076994717121 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_6‚1 particle/accuracy_7_of_7‚0.5 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3392255902290344 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967495203018188 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07446808367967606 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9600694179534912 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5768178105354309 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973205924034119 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1784611940383911 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961740970611572 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5593422055244446 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5089285373687744 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4553571343421936 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3303303122520447 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9725422263145447 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4645901024341583 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9449564218521118 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9847952723503113 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07824164628982544 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05119047686457634 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2369121015071869 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21459786593914032 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12448559701442719 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21905478835105896 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.52521479129791vïu226 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4185529947280884 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3565877079963684 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4385965168476105 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.2563016414642334 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2167578488588333 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5486452579498291 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08376705646514893 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11100342124700546 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2120632529258728 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013235841877758503 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43869107961654663 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05962643772363663 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9868838787078857 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4217284917831421 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.20577600598335266 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 jet/accuracy_5_of_6‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43800318241119385 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9850649833679199 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9857777953147888 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4801599383354187 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02950832061469555 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25474613904953003 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4118509888648987 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2916020452976227 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07522246986627579 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4281609356403351 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4337919056415558 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010355253703892231 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.521812379360199 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5511854887008667 jet/accuracy_5_of_7‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.43097352981567383 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39498910307884216 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.26591548323631287 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08740007877349854 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.43778344988822937 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0173611119389534 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22521944344043732 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022741014137864113 particle/accuracy_2_of_2‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.08974359929561615 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4904961884021759 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2103249877691269 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009542534127831459 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3482142984867096 particle/accuracy_4_of_4‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3541666865348816 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5267360806465149 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9922136068344116 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2807539403438568 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16649426519870758 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4638778567314148 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3580247163772583 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 _step‚395 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2905172109603882 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +jet/accuracy_3_of_3‚0.01627030223608017 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_1_of_7‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45600420236587524 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9834979772567749 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4544544219970703 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4499461054801941 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.095238097012043 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16872799396514893 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18114754557609558 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_1_of_5‚1 jet/accuracy_1_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4451117515563965 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9874655604362488 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37733912467956543 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.578696072101593 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.48279279470443726 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.446422278881073 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19933390617370605 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8356326222419739 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5196883678436279 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14240966737270355 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5034722089767456 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 particle/accuracy_4_of_7‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39048007130622864 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5133333206176758 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9884737730026245 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.37179484963417053 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.021278882399201393 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023362306877970695 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.0957864448428154 particle/accuracy_3_of_5‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_2_of_5‚0.38903605937957764 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3601851463317871 particle/accuracy_5_of_6‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17411589622497559 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4597829580307007 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9822197556495667 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9975579977035522 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224‹@œƒ‚(:‘ …åÔűö gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.750200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»¹íÙ$„‚(:™ …åÔñ¼ó) disk./.usagePercent‚73.87490500072418 disk.dm-0.in‚ 139.9296875 disk.rbd0.in‚0# memory_percent‚98.0797290802002 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.dm-0.out‚20407.12890625 disk.dm-1.out‚ 146.84765625 disk.sda3.in‚0 disk.sda.out‚ 146.84765625 disk.sda1.in‚0% disk.nvme0c0n1.out‚20407.12890625 disk.sda2.out‚0 disk.rbd2.in‚0 cpu‚25.370467679524797 disk.sda1.out‚0# disk.nvme0n1.out‚20407.12890625 disk.sda2.in‚0! disk.rbd2.out‚117.52685546875 disk.sda3.out‚ 146.84765625 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 139.9296875 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 3030.48828125 network.recv‚ 1.34385e+06 disk.nvme0n1.in‚ 139.9296875 disk.rbd1.in‚0) proc.memory.percent‚18.49663257598877% disk./.usageGB‚1100.7894706726074 disk.dm-1.in‚0 disk.rbd1.out‚ 1.83984375' proc.memory.availableMB‚ 314.6171875 network.sent‚ 1.0633796e+07è¥`2…- _wandbruntime‚3702  _runtime‚3702 í3Ɇà ' lr-AdamW/pg1‚0.00016959838396746841 ' lr-AdamW/pg2‚0.00016959838396746841 trainer/global_step‚4849 % _timestamp‚1.7884320181444328e+09 _runtime‚3702.849165589 _step‚396•ÊÍÎ‡È _runtime‚3702.849165589 _step‚396 ' lr-AdamW/pg1‚0.00016959838396746841 ' lr-AdamW/pg2‚0.00016959838396746841 trainer/global_step‚4849 % _timestamp‚1.7884320181444328e+09Œjàé2ˆ- _wandbruntime‚3702  _runtime‚3702t±Ë®Ý'‰×O YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚279247889303760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70708308 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4266198818816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4052473831252 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286678426648064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41259508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚282610300153344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23297226 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚275194761034752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286677752282624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚4849 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91689348 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70708310 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282610300153344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308571393884672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91689348 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286678426648064 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282610318156288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70708308 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚275194743995392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚312837616643584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91669310 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279247217826644 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4067452129280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286677752282624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚3702.850021432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23296778 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64556736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162397658 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚312837429079552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚308571417824768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4053128269008 % _timestamp‚1.7884320181451318e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23297226 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚275194761034752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286677752282624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚4849 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91689348 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70708310 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282610300153344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308571393884672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91689348 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286678426648064 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282610318156288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70708308 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚275194743995392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_bat%MãÐ. ch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚312837616643584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91669310 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279247217826644 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4067452129280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286677752282624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚3702.850021432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23296778 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430ÎœF2‹- _wandbruntime‚3703  _runtime‚3703ÏÃS±ç&ŒáM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3703.515023176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4053944704180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64568953 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚308633044622336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275249648125952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282666653192192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91707828 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162410088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279303609869492 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286734924749312 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚312899938982400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91687764 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4068271557120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚398 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162430154 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884320188094726e+09 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4053290266216 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162430154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70722326 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4068927932928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64569395 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312900126533120 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70722326 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275249665165312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41267676 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23301717 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚4849 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162410088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91707828 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282666671195136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282666671195136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚282666653192192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4068927932928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4267057970688 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884320188094726e+09 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4053290266216 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162430154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70722326 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4068927932928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64569395 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312900126533120 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70722326 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275249665165312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41267676 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23301717 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚4849 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162410088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91707828 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282666671195136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282666671195136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚282666653192192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4068927932928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4267057970688 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3703.515023176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4053944704180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64568953 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚308633044622336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275249648125952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282666653192192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91707828 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162410088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279303609869492 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286734924749312 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚312899938982400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91687764 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4068271557120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚398 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ŽR»™2Ž- _wandbruntime‚3703  _runtime‚3703>ïÑ+]× $ _timestamp‚1.788432018811463e+09 /loss/FRt2/assignment_loss‚2.917218918049777 /loss/FRt4/detection_loss‚2.0460043200906717 1loss/SRqqt3/detection_loss‚2.6146706671550355 1loss/SRqqt1/detection_loss‚1.0212487556406709 _runtime‚3703.516101177 3loss/FBt2/assignment_loss‚0.0063854766776785254 epoch‚4 trainer/global_step‚4849 0loss/FRt3/assignment_loss‚3.8585612092699324 0loss/FBt3/assignment_loss‚0.4001790373586118 %loss/total_loss‚45.29791580426502 /loss/FRt1/assignment_loss‚2.082612709123261 1loss/FBt1/assignment_loss‚0.21032901858491376 /loss/FRt2/detection_loss‚2.3732764417374574 0loss/SRqqt2/detection_loss‚1.924681009764367 1loss/SRqqt4/detection_loss‚1.7216000362318389 2loss/SRqqt3/assignment_loss‚1.7208955598802402 /loss/FBt1/detection_loss‚0.5725811204761541 _step‚399 2loss/SRqqt4/assignment_loss‚1.6347155781576828 1loss/FBt4/assignment_loss‚0.38455886448500676 .loss/FBt2/detection_loss‚3.612921857833862 /loss/FBt4/detection_loss‚2.4029962003231047 0loss/FRt4/assignment_loss‚3.3232706976386734 2loss/SRqqt1/assignment_loss‚0.9130441151925774 2loss/SRqqt2/assignment_loss‚1.4403008680711402 .loss/FRt1/detection_loss‚2.009609331129765 /loss/FRt3/detection_loss‚2.9840044932706014 .loss/FBt3/detection_loss‚3.122249518121992ÈÎdbÜ 2loss/SRqqt4/assignment_loss‚1.6347155781576828 1loss/FBt4/assignment_loss‚0.38455886448500676 .loss/FBt2/detection_loss‚3.612921857833862 /loss/FBt4/detection_loss‚2.4029962003231047 0loss/FRt4/assignment_loss‚3.3232706976386734 2loss/SRqqt1/assignment_loss‚0.9130441151925774 2loss/SRqqt2/assignment_loss‚1.4403008680711402 .loss/FRt1/detection_loss‚2.009609331129765 /loss/FRt3/detection_loss‚2.9840044932706014 .loss/FBt3/detection_loss‚3.122249518121992 $ _timestamp‚1.788432018811463e+09 /loss/FRt2/assignment_loss‚2.917218918049777 /loss/FRt4/detection_loss‚2.0460043200906717 1loss/SRqqt3/detection_loss‚2.6146706671550355 1loss/SRqqt1/detection_loss‚1.0212487556406709 _runtime‚3703.516101177 3loss/FBt2/assignment_loss‚0.0063854766776785254 epoch‚4 trainer/global_step‚4849 0loss/FRt3/assignment_loss‚3.8585612092699324 0loss/FBt3/assignment_loss‚0.4001790373586118 %loss/total_loss‚45.29791580426502 /loss/FRt1/assignment_loss‚2.082612709123261 1loss/FBt1/assignment_loss‚0.21032901858491376 /loss/FRt2/detection_loss‚2.3732764417374574 0loss/SRqqt2/detection_loss‚1.924681009764367 1loss/SRqqt4/detection_loss‚1.7216000362318389 2loss/SRqqt3/assignment_loss‚1.7208955598802402 /loss/FBt1/detection_loss‚0.5725811204761541 _step‚399VSŽž‘‚(:“ ”åÔšŠ„Ž gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚103.261' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.652199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PÔA$’‚(:™ ”åÔõ× disk.sda2.out‚0! disk.rbd2.out‚117.53466796875 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda.in‚0) proc.memory.percent‚17.53101348876953 disk.rbd0.out‚0 disk.sda1.in‚0$ memory_percent‚91.30022525787354$ disk.nvme0n1.out‚21668.650390625 disk.rbd2.in‚0 disk.sda3.out‚ 147.49609375 cpu‚25.36864616826866 proc.cpu.threads‚17 network.recv‚ 1.357162e+06 disk.rbd1.in‚0! disk.dm-0.out‚21668.650390625 disk.sda.out‚ 147.49609375 proc.memory.rssMB‚ 2872.28125 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0( disk./.usagePercent‚73.8748960404605 disk.dm-1.in‚0 disk.dm-1.out‚ 147.49609375 disk.rbd1.out‚ 1.83984375! disk.nvme0c0n1.in‚ 139.9296875& disk.nvme0c0n1.out‚21668.650390625 disk.nvme0n1.in‚ 139.9296875 disk.sda2.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 139.9296875) proc.memory.availableMB‚ 1425.37109375 network.sent‚ 1.07819e+07% disk./.usageGB‚1100.7893371582031E„j““‚(:ˆ £åÔñ‘‚Ž gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚161.186' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0  4(”‚(: £åÔå¿ú network.sent‚ 1.0897875e+07( disk.nvme0c0n1.out‚21768.35009765625# disk.dm-0.out‚21768.35009765625 proc.memory.rssMB‚ 2872.34375$ disk./.usageGB‚1100.789348602295 disk.nvme0n1.in‚ 139.9296875 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 148.04296875 disk.sda3.in‚0 disk.sda.out‚ 148.04296875# memory_percent‚91.3083553314209 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 1424.0390625 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 139.9296875 disk.sda2.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 148.04296875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 139.9296875 network.recv‚ 1.378628e+06( disk./.usagePercent‚73.8748968084831& disk.nvme0n1.out‚21768.35009765625 disk.rbd0.in‚0 disk.rbd2.out‚ 117.62109375 disk.rbd3.in‚0* proc.memory.percent‚17.531394958496094 cpu‚25.367313796047835 disk.rbd1.out‚ 1.83984375 disk.dm-1.in‚0 proc.cpu.threads‚17áâû»“•‚(:ˆ ²åÔÂŒì gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÇŒn–‚(:ˆ ²åÔ»„¢( disk./.usagePercent‚73.8748968084831 disk.sda.in‚0( disk.nvme0c0n1.out‚21768.62744140625 disk.rbd0.out‚0$ memory_percent‚91.30067825317383 proc.memory.rssMB‚ 2872.34375 network.sent‚ 1.0906077e+07 disk.drbd1777.in‚0 disk.dm-0.in‚ 139.9296875# disk.dm-0.out‚21768.62744140625 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 117.63671875 proc.cpu.threads‚17 disk.sda3.out‚148.5& disk.nvme0n1.out‚21768.62744140625 cpu‚25.366043861442066 disk.sda.out‚148.5 disk.nvme0n1.in‚ 139.9296875! disk.nvme0c0n1.in‚ 139.9296875 disk.rbd0.in‚0* proc.memory.percent‚17.531394958496094 network.recv‚ 1.388421e+06 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚148.5 disk.sda2.in‚0' proc.memory.availableMB‚ 1425.296875$ disk./.usageGB‚1100.789348602295 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 1.83984375Be62—- _wandbruntime‚3737  _runtime‚3737G ŽÇ˜Á & lr-AdamW/pg1‚0.0001681728819852878 & lr-AdamW/pg2‚0.0001681728819852878 trainer/global_step‚4899 % _timestamp‚1.7884320531136003e+09 _runtime‚3737.818300011 _step‚400æâhÌ™Æ & lr-AdamW/pg1‚0.0001681728819852878 & lr-AdamW/pg2‚0.0001681728819852878 trainer/global_step‚4899 % _timestamp‚1.7884320531136003e+09 _runtime‚3737.818300011 _step‚400À‘DH2š- _wandbruntime‚3737  _runtime‚3737ú— UD ›×O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277939967562752 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚285427970098688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92595362 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277939950523392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4093963608416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71409108 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65191030 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4093309170660 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315963067368960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282033259694052 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚4899 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65191459 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289537064396288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚285427952095w ž™744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71409110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚3737.819298091 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289536390030848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4108437935104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92615400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884320531143754e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚285427952095744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23523549 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285427970098688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4309136601088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41667908 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4109094297600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311653930767872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289536390030848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41667910 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289537064396288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4109094297600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4309300225024 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71409110 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23523122 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164004470 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71409108 _step‚401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚311653954707968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92615400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884320531143754e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚285427952095744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23523549 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285427970098688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4309136601088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41667908 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4109094297600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311653930767872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289536390030848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41667910 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289537064396288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4109094297600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4309300225024 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71409110 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23523122 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164004470 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71409108 _step‚401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚311653954707968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277939967562752 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚285427970098688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92595362 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277939950523392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4093963608416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71409108 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65191030 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4093309170660 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315963067368960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282033259694052 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚4899 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65191459 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289537064396288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚285427952095744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71409110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872‘‹=¾2- _wandbruntime‚3738  _runtime‚3738‰šæ&žàM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289594236876288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41676076 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92613816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311715605445632 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164036940 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65204058 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109257362944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282088980259576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71423124 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285484305134592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23527980 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884320537676935e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289593562497536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚285484305134592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164057006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚4899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71423126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285484323137536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4094125605624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164057006 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71423126 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92613816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4109257362944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316025763784192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4109913738752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4310158338560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4109913738752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23527545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚311715581505536 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277994871693312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4094780043588 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚402 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71423124 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41676078 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164036940 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92633880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65203621 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277994854653952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 6544379645Œ³rJŸåM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4109913738752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4310158338560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4109913738752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23527545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚311715581505536 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277994871693312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4094780043588 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚402 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71423124 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41676078 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164036940 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92633880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65203621 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277994854653952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289594236876288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41676076 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92613816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311715605445632 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164036940 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65204058 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109257362944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/\Z+i¡requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282088980259576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71423124 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285484305134592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23527980 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884320537676935e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289593562497536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚285484305134592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164057006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚4899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71423126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285484323137536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4094125605624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164057006 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71423126 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92613816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4109257362944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316025763784192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360’Ï«ó²2 - _wandbruntime‚3738  _runtime‚3738rLÑZ¡Ô 0loss/FBt3/assignment_loss‚0.6079522857326083 1loss/SRqqt1/detection_loss‚1.1717955138836358 0loss/SRqqt3/detection_loss‚2.532259974628687 _runtime‚3738.473958339 /loss/FRt3/detection_loss‚2.6568372910087175 .loss/FBt2/detection_loss‚2.321179561316967 &loss/total_loss‚44.331956126875056 _step‚403 /loss/FRt4/assignment_loss‚3.319875293307834 1loss/FBt4/assignment_loss‚0.09923491839851652 0loss/SRqqt4/detection_loss‚1.964084888306948 /loss/FBt3/detection_loss‚2.7458106756210325 /loss/FRt2/assignment_loss‚2.544161478226835 2loss/SRqqt3/assignment_loss‚1.8551996015012264 0loss/SRqqt2/detection_loss‚1.738281352792756 2loss/SRqqt4/assignment_loss‚1.4934896253264294 /loss/FRt1/detection_loss‚1.8337276358993686 trainer/global_step‚4899 % _timestamp‚1.7884320537693512e+09 0loss/FRt3/assignment_loss‚2.9375368743329435 2loss/SRqqt1/assignment_loss‚1.2131885385417656 2loss/SRqqt2/assignment_loss‚1.7761600859842057 1loss/FBt2/assignment_loss‚0.29452211373427417 .loss/FBt1/detection_loss‚0.898546740521345 /loss/FBt4/detection_loss‚3.7399745413235257 0loss/FRt1/assignment_loss‚1.5732983606202262 /loss/FRt2/detection_loss‚2.3855678459460083 epoch‚4 0loss/FBt1/assignment_loss‚0.2248842717805039 .loss/FRt4/detection_loss‚2.404386658138699C:ž_¢Ù % _timestamp‚1.7884320537693512e+09 0loss/FRt3/assignment_loss‚2.9375368743329435 2loss/SRqqt1/assignment_loss‚1.2131885385417656 2loss/SRqqt2/assignment_loss‚1.7761600859842057 1loss/FBt2/assignment_loss‚0.29452211373427417 .loss/FBt1/detection_loss‚0.898546740521345 /loss/FBt4/detection_loss‚3.7399745413235257 0loss/FRt1/assignment_loss‚1.5732983606202262 /loss/FRt2/detection_loss‚2.3855678459460083 epoch‚4 0loss/FBt1/assignment_loss‚0.2248842717805039 .loss/FRt4/detection_loss‚2.404386658138699 0loss/FBt3/assignment_loss‚0.6079522857326083 1loss/SRqqt1/detection_loss‚1.1717955138836358 0loss/SRqqt3/detection_loss‚2.532259974628687 _runtime‚3738.473958339 /loss/FRt3/detection_loss‚2.6568372910087175 .loss/FBt2/detection_loss‚2.321179561316967 &loss/total_loss‚44.331956126875056 _step‚403 /loss/FRt4/assignment_loss‚3.319875293307834 1loss/FBt4/assignment_loss‚0.09923491839851652 0loss/SRqqt4/detection_loss‚1.964084888306948 /loss/FBt3/detection_loss‚2.7458106756210325 /loss/FRt2/assignment_loss‚2.544161478226835 2loss/SRqqt3/assignment_loss‚1.8551996015012264 0loss/SRqqt2/detection_loss‚1.738281352792756 2loss/SRqqt4/assignment_loss‚1.4934896253264294 /loss/FRt1/detection_loss‚1.8337276358993686 trainer/global_step‚4899“A5^vž£‚(:“ ÁåÔñåç gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.658' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.531599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØÓår!¤‚(:– ÁåÔ¹ñ¸ disk.rbd3.out‚0 network.sent‚ 1.0916245e+07 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda.out‚ 149.11328125) disk./.usagePercent‚73.87489988057351 disk.rbd1.out‚ 1.92578125( disk.nvme0c0n1.out‚21868.61376953125 disk.rbd2.out‚ 117.76171875 disk.sda3.out‚ 149.11328125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 cpu‚25.36478587641476 disk.sda1.out‚0 disk.dm-0.in‚139.9375$ disk./.usageGB‚1100.789394378662 disk.sda1.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚139.9375 disk.sda2.in‚0* proc.memory.percent‚17.531394958496094# disk.dm-0.out‚21868.61376953125 disk.dm-1.out‚ 149.11328125& disk.nvme0n1.out‚21868.61376953125 disk.rbd0.out‚0 disk.sda3.in‚0 proc.memory.rssMB‚ 2872.34375 disk.nvme0c0n1.in‚139.9375 proc.cpu.threads‚17 network.recv‚ 1.400902e+06 disk.rbd3.in‚0$ memory_percent‚91.30313396453857) proc.memory.availableMB‚ 1424.89453125Ëã Ö“¥‚(:ˆ ÐåÔ‹¡†Ž gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qn»[¦‚(:” ÐåÔý•œ‘% disk./.usageGB‚1100.7889938354492 disk.dm-0.in‚139.9375 disk.sda1.in‚0 network.sent‚ 1.1027553e+07 disk.sda3.in‚0 disk.dm-1.out‚ 149.66796875 disk.rbd1.out‚ 1.98828125( disk.nvme0c0n1.out‚21967.70361328125' proc.memory.availableMB‚ 1424.921875 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.531394958496094) disk./.usagePercent‚73.87487299978241 disk.rbd2.in‚0 disk.sda3.out‚ 149.66796875 disk.rbd1.in‚0 disk.nvme0c0n1.in‚139.9375 network.recv‚ 1.413219e+06 disk.rbd2.out‚ 119.2890625# disk.dm-0.out‚21967.70361328125 disk.sda2.out‚0# memory_percent‚91.3029670715332 proc.memory.rssMB‚ 2872.34375 cpu‚25.363529689188088 proc.cpu.threads‚17 disk.nvme0n1.in‚139.9375 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 149.66796875 disk.sda1.out‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚21967.70361328125«âÌi2§- _wandbruntime‚3772  _runtime‚3772'M–WȨ _step‚404 ' lr-AdamW/pg1‚0.00016673594695318944 ' lr-AdamW/pg2‚0.00016673594695318944 trainer/global_step‚4949 $ _timestamp‚1.788432088102585e+09 _runtime‚3772.8072962085â EÍ©Ç trainer/global_step‚4949 $ _timestamp‚1.788432088102585e+09 _runtime‚3772.807296208 _step‚404 ' lr-AdamW/pg1‚0.00016673594695318944 ' lr-AdamW/pg2‚0.00016673594695318944”)€2ª- _wandbruntime‚3772  _runtime‚3772/‘ÎÝ'«×O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280685174090752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚288245622041088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72109908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4134798947824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4134144510068 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72109908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292395027779072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319088684844032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93541452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23750882 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23750447 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284819973038576 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72109910 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93521414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65826755 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚405 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4150080103424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280685157051392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292395702144512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165651362 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314736467651072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884320881032553e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚288245604038144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4149423740928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚4949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165631322 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42076308 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284819301561460 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚3772.808176287j×ââ'¬ÜO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72109910 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93521414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65826755 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚405 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4150080103424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280685157051392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292395702144512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165651362 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314736467651072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884320881032553e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚288245604038144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4149423740928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚4949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165631322 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42076308 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284819301561460 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚3772.808176287 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280685174090752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚288245622041088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72109908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4134798947824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4134144510068 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72109908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292395027779072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319088684844032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93541452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23750882 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23750447 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284819973038576•ÜŠ2­- _wandbruntime‚3773  _runtime‚3773†0r©®áM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288301957076992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93559932 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4134960945032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large¯FÁ>$_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72123926 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚3773.459800785 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4353236871168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93539868 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288301957076992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65839928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65839489 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314798118388736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23755450 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319151379200000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314798142328832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42084478 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280740061181952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292452874624512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280740078221312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚288301975079936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4150899544576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292452874624512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4150243168768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4353069066240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4135615382996 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4949 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93539868 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93559932 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165683858 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844"dÕì&¯æM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42084478 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280740061181952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292452874624512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280740078221312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚288301975079936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4150899544576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292452874624512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4150243168768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4353069066240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4135615382996 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4949 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93539868 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93559932 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165683858 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288301957076992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93559932 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4134960945032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72123926 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚3773.459800785 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4353236871168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93539868 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288301957076992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65839928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65839489 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314798118388736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23755450 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319151379200000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314798142328832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414–ýôyù2°- _wandbruntime‚3773  _runtime‚3773µN5`±Ú 1loss/SRqqt2/detection_loss‚1.9898760827879112 % _timestamp‚1.7884320887560527e+09 1loss/FBt4/assignment_loss‚0.24507745107014975 0loss/FRt2/assignment_loss‚3.1476069276149454 /loss/FRt4/assignment_loss‚4.081744521856308 2loss/SRqqt2/assignment_loss‚1.4911823129902284 /loss/FRt4/detection_loss‚2.3647831169267497 .loss/FRt1/detection_loss‚1.798991065914348 .loss/FRt3/detection_loss‚3.078031365076701 1loss/SRqqt1/detection_loss‚1.0943020720979277 1loss/SRqqt4/detection_loss‚2.2084847105873955 2loss/SRqqt1/assignment_loss‚1.1837192678850301 2loss/SRqqt4/assignment_loss‚1.8287755032380422 4loss/FBt2/assignment_loss‚0.00044023370719514787 0loss/FRt3/assignment_loss‚3.8541139721870423 1loss/FBt1/assignment_loss‚0.15629335882454926 2loss/FBt3/assignment_loss‚0.023668335440258186 /loss/FBt1/detection_loss‚0.5894006953185683 /loss/FBt4/detection_loss‚1.9771942959891424 &loss/total_loss‚46.699664070516306 /loss/SRqqt3/detection_loss‚2.21275187370389 trainer/global_step‚4949 -loss/FRt2/detection_loss‚2.74186179958857 0loss/FRt1/assignment_loss‚1.7051353313155093 1loss/SRqqt3/assignment_loss‚1.818789801452049 _runtime‚3773.460756827 /loss/FBt2/detection_loss‚3.0631229281425476 /loss/FBt3/detection_loss‚4.0443170468012495 epoch‚4 _step‚407²¤—te²ß /loss/FBt2/detection_loss‚3.0631229281425476 /loss/FBt3/detection_loss‚4.0443170468012495 epoch‚4 _step‚407 1loss/SRqqt2/detection_loss‚1.9898760827879112 % _timestamp‚1.7884320887560527e+09 1loss/FBt4/assignment_loss‚0.24507745107014975 0loss/FRt2/assignment_loss‚3.1476069276149454 /loss/FRt4/assignment_loss‚4.081744521856308 2loss/SRqqt2/assignment_loss‚1.4911823129902284 /loss/FRt4/detection_loss‚2.3647831169267497 .loss/FRt1/detection_loss‚1.798991065914348 .loss/FRt3/detection_loss‚3.078031365076701 1loss/SRqqt1/detection_loss‚1.0943020720979277 1loss/SRqqt4/detection_loss‚2.2084847105873955 2loss/SRqqt1/assignment_loss‚1.1837192678850301 2loss/SRqqt4/assignment_loss‚1.8287755032380422 4loss/FBt2/assignment_loss‚0.00044023370719514787 0loss/FRt3/assignment_loss‚3.8541139721870423 1loss/FBt1/assignment_loss‚0.15629335882454926 2loss/FBt3/assignment_loss‚0.023668335440258186 /loss/FBt1/detection_loss‚0.5894006953185683 /loss/FBt4/detection_loss‚1.9771942959891424 &loss/total_loss‚46.699664070516306 /loss/SRqqt3/detection_loss‚2.21275187370389 trainer/global_step‚4949 -loss/FRt2/detection_loss‚2.74186179958857 0loss/FRt1/assignment_loss‚1.7051353313155093 1loss/SRqqt3/assignment_loss‚1.818789801452049 _runtime‚3773.460756827—Ä轞³‚(:“ ßåÔõ÷ò gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.587' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.517399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃÈUb"´‚(:— ßåÔ“‚Ê disk.rbd3.in‚0 cpu‚25.362302773813003 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 1.98828125 disk.nvme0c0n1.in‚139.9375( disk.nvme0c0n1.out‚22066.88818359375 disk.rbd3.out‚0 disk.nvme0n1.in‚139.9375& disk.nvme0n1.out‚22066.88818359375 disk.rbd2.out‚ 119.4140625 disk.sda3.out‚ 150.21484375 disk.drbd1777.in‚0$ memory_percent‚91.30356311798096% disk./.usageGB‚1100.7889938354492# disk.dm-0.out‚22066.88818359375 disk.sda.in‚0 disk.sda.out‚ 150.21484375 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.531394958496094 disk.dm-1.out‚ 150.21484375 proc.memory.rssMB‚ 2872.34375 proc.cpu.threads‚17 network.recv‚ 1.417885e+06 disk.dm-0.in‚139.9375 disk.sda1.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1424.82421875 network.sent‚ 1.1033809e+07) disk./.usagePercent‚73.87487299978241 disk.rbd0.in‚0æ 2“µ‚(:ˆ îåÔó‚ë gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.083' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"K Y ¶‚(:• îåÔü‘† network.recv‚ 1.423475e+06 disk.drbd1777.out‚0 disk.nvme0n1.in‚139.9375 disk.rbd1.in‚0& proc.memory.availableMB‚ 1424.90625 disk.rbd3.in‚0 disk.dm-0.in‚139.9375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 150.71484375$ memory_percent‚91.30306243896484 network.sent‚ 1.1141671e+07 disk.rbd3.out‚0 disk.sda.out‚ 150.71484375 disk.sda1.out‚0( disk.nvme0c0n1.out‚22165.98583984375 disk.sda2.in‚0* proc.memory.percent‚17.531394958496094 disk.dm-1.in‚0 disk.dm-1.out‚ 150.71484375& disk.nvme0n1.out‚22165.98583984375 disk.rbd1.out‚ 2.0205078125 disk.rbd2.out‚ 119.4296875 proc.memory.rssMB‚ 2872.34375) disk./.usagePercent‚73.87487683989544 proc.cpu.threads‚17# disk.dm-0.out‚22165.98583984375 disk.rbd0.in‚0 disk.sda3.in‚0% disk./.usageGB‚1100.7890510559082 disk.drbd1777.in‚0 disk.sda.in‚0 disk.nvme0c0n1.in‚139.9375 cpu‚25.36101867071653 disk.sda2.out‚0 disk.rbd2.in‚0¤Uó[2·- _wandbruntime‚3807  _runtime‚3807Zwy@ɸà % _timestamp‚1.7884321228360958e+09 _runtime‚3807.540849299 _step‚408 ' lr-AdamW/pg1‚0.00016528787832628043 ' lr-AdamW/pg2‚0.00016528787832628043 trainer/global_step‚4999±ªÎÎ¹È ' lr-AdamW/pg1‚0.00016528787832628043 ' lr-AdamW/pg2‚0.00016528787832628043 trainer/global_step‚4999 % _timestamp‚1.7884321228360958e+09 _runtime‚3807.540849299 _step‚408˜&¡¿%2º- _wandbruntime‚3807  _runtime‚3807ÇI­b»×O QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚287606014905984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42484708 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚295253665527296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72810708 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72810708 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4395550673408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚287605343428868 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23978621 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317819004534272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚409 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167258174 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚295254339892736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295253665527296 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚291063255980544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4190409546752 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94467504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚291063255980544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283430380618752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66463329 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚322214395778048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚4999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚291063273983488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322214579147776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚291063273983488 UGDeviceStatsMonitor.on_train_batch_ä*v%Î start/allocated_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4191065909248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4395391243776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72810710 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317819028474368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42484710 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚295254339892736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94467504 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚3807.541811292 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94447466 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884321228368278e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0gVâ'¼ÜO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4395550673408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚287605343428868 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23978621 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317819004534272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚409 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167258174 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚295254339892736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295253665527296 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚291063255980544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4190409546752 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94467504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚291063255980544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283430380618752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66463329 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚322214395778048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚4999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚291063273983488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322214579147776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚291063273983488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4191065909248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4395391243776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72810710 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317819028474368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42484710 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚295254339892736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94467504 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚3807.541811292 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94447466 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884321228368278e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚287606014905984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42484708 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚295253665527296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72810708 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72810708 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889™m õÓ2½- _wandbruntime‚3808  _runtime‚3808åHÔç&¾áM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72824724 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167310710 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72824726 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291119609019392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291119609019392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167290644 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94465920 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚291119627022336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291119627022336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72824726 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884321234924173e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4191228974592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283485267709952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚295311512372736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322276912654336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚317880655271936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4175796284440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66475945 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚295311512372736 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167310710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42492876 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283485284749312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322277096010752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4396416798720 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚410 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4191885350400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94485984 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72824724 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94485984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4176450722404 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844hk÷ý%¿æM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291119609019392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291119609019392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167290644 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94465920 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚291119627022336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291119627022336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72824726 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884321234924173e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4191228974592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283485267709952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚295311512372736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322276912654336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚317880655271936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4175796284440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66475945 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚295311512372736 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167310710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42492876 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283485284749312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322277096010752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4396416798720 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚410 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4191885350400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94485984 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72824724 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94485984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4176450722404 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72824724 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167310710 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADevëyØiceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72824726 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2šàdÐ2À- _wandbruntime‚3808  _runtime‚3808(~¦3WÁÑ 1loss/SRqqt1/detection_loss‚0.9649814447682155 .loss/FBt3/detection_loss‚4.035318523645401 /loss/FRt2/detection_loss‚2.5503368200290772 /loss/FRt4/assignment_loss‚3.627658088098873 1loss/SRqqt3/assignment_loss‚1.798250937932416 1loss/FBt2/assignment_loss‚0.19520488717292514 2loss/SRqqt4/assignment_loss‚1.5081054546288215 1loss/FBt1/assignment_loss‚0.15440868641285257 trainer/global_step‚4999 2loss/SRqqt1/assignment_loss‚1.0525795467393964 /loss/FBt2/detection_loss‚2.2017826437950134 epoch‚4 1loss/SRqqt2/assignment_loss‚1.575249124189903 1loss/SRqqt3/detection_loss‚2.3316525346354435 1loss/SRqqt4/detection_loss‚2.1651196725666524 /loss/FBt1/detection_loss‚0.6553348056259376 /loss/FBt4/detection_loss‚2.2536669288362776 _runtime‚3808.199124059 1loss/FBt3/assignment_loss‚0.08895137079889537 /loss/FBt4/assignment_loss‚0.522080554493836 .loss/FRt1/detection_loss‚1.652607719508969 /loss/FRt2/assignment_loss‚3.266904202245531 .loss/FRt3/detection_loss‚3.016147257760167 1loss/SRqqt2/detection_loss‚1.8482250157615234 _step‚411 /loss/FRt3/assignment_loss‚3.391917334869504 % _timestamp‚1.7884321234944737e+09 %loss/total_loss‚44.01518224425418 0loss/FRt1/assignment_loss‚1.0526348036162707 -loss/FRt4/detection_loss‚2.106063886122278#U\ÂÖ % _timestamp‚1.7884321234944737e+09 %loss/total_loss‚44.01518224425418 0loss/FRt1/assignment_loss‚1.0526348036162707 -loss/FRt4/detection_loss‚2.10606388612227 1loss/SRqqt1/detection_loss‚0.9649814447682155 .loss/FBt3/detection_loss‚4.035318523645401 /loss/FRt2/detection_loss‚2.5503368200290772 /loss/FRt4/assignment_loss‚3.627658088098873 1loss/SRqqt3/assignment_loss‚1.798250937932416 1loss/FBt2/assignment_loss‚0.19520488717292514 2loss/SRqqt4/assignment_loss‚1.5081054546288215 1loss/FBt1/assignment_loss‚0.15440868641285257 trainer/global_step‚4999 2loss/SRqqt1/assignment_loss‚1.0525795467393964 /loss/FBt2/detection_loss‚2.2017826437950134 epoch‚4 1loss/SRqqt2/assignment_loss‚1.575249124189903 1loss/SRqqt3/detection_loss‚2.3316525346354435 1loss/SRqqt4/detection_loss‚2.1651196725666524 /loss/FBt1/detection_loss‚0.6553348056259376 /loss/FBt4/detection_loss‚2.2536669288362776 _runtime‚3808.199124059 1loss/FBt3/assignment_loss‚0.08895137079889537 /loss/FBt4/assignment_loss‚0.522080554493836 .loss/FRt1/detection_loss‚1.652607719508969 /loss/FRt2/assignment_loss‚3.266904202245531 .loss/FRt3/detection_loss‚3.016147257760167 1loss/SRqqt2/detection_loss‚1.8482250157615234 _step‚411 /loss/FRt3/assignment_loss‚3.391917334869504›Ǥ1ޜÂ(:‘ ýåÔ÷êÙŽ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.26' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.251999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…Ø’!Ä‚(:– ýåÔñ¯· disk.drbd1777.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 cpu‚25.359797157663433 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-0.in‚139.9375% disk.nvme0n1.out‚22166.6162109375 disk.nvme0c0n1.in‚139.9375! disk.rbd2.out‚119.57861328125* proc.memory.percent‚17.531394958496094% disk./.usageGB‚1100.7890319824219 disk.rbd3.out‚0 disk.sda1.out‚0$ memory_percent‚91.30609035491943 proc.memory.rssMB‚ 2872.34375 network.recv‚ 1.428371e+06) disk./.usagePercent‚73.87487555985776" disk.dm-0.out‚22166.6162109375) proc.memory.availableMB‚ 1424.41015625 disk.nvme0n1.in‚139.9375 disk.sda2.out‚0 network.sent‚ 1.114793e+07 disk.sda.in‚0 disk.sda.out‚ 151.2734375 disk.dm-1.out‚ 151.2734375 proc.cpu.threads‚17' disk.nvme0c0n1.out‚22166.6162109375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 151.2734375 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 2.0205078125 disk.rbd2.in‚0ˆ ¯´žÅ‚(:“ ŒžåÔ®Þû gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚111.772' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.354400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R #Æ‚(:˜ ŒžåÔÈŽÆ disk.rbd1.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 cpu‚25.35865388058795) disk./.usagePercent‚73.87488144803105$ disk./.usageGB‚1100.789119720459 disk.sda.out‚ 151.7890625 proc.memory.rssMB‚ 2872.34375 disk.sda2.out‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1424.66015625 proc.cpu.threads‚17! disk.rbd2.out‚119.59423828125 disk.rbd3.out‚0 disk.nvme0c0n1.in‚139.9375 network.recv‚ 1.435272e+06 disk.sda3.in‚0 disk.sda3.out‚ 151.7890625 disk.dm-0.in‚139.9375# disk.dm-0.out‚22265.80712890625 disk.nvme0n1.in‚139.9375 disk.rbd0.out‚0 network.sent‚ 1.1256415e+07 disk.rbd3.in‚0 disk.dm-1.out‚ 151.7890625& disk.nvme0n1.out‚22265.80712890625 disk.rbd1.out‚ 2.0205078125$ memory_percent‚91.30456447601318 disk.drbd1777.in‚0( disk.nvme0c0n1.out‚22265.80712890625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.531394958496094¹,z‘Ç‚(:† ›žåÔ®„Û gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚107.46' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Ó È‚(:• ›žåÔŸíë disk.rbd3.in‚0" disk.dm-0.out‚22365.4501953125 disk.sda.out‚ 152.3359375 disk.dm-1.in‚0 disk.dm-1.out‚ 152.3359375' disk.nvme0c0n1.out‚22365.4501953125 disk.sda2.out‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚22365.4501953125 disk.nvme0c0n1.in‚139.9375) disk./.usagePercent‚73.87488144803105$ disk./.usageGB‚1100.789119720459 disk.dm-0.in‚139.9375 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 2.08349609375$ memory_percent‚91.30449295043945* proc.memory.percent‚17.531394958496094 network.recv‚ 1.440011e+06' proc.memory.availableMB‚ 1424.671875 cpu‚25.357387753764527 disk.rbd3.out‚0 disk.nvme0n1.in‚139.9375 disk.sda2.in‚0! disk.rbd2.out‚121.16845703125 disk.drbd1777.in‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 2872.34375 proc.cpu.threads‚17 network.sent‚ 1.1262768e+07 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 152.3359375 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0Œ*Ž?2É- _wandbruntime‚3842  _runtime‚3842r?²ÉÊà _step‚412 ' lr-AdamW/pg1‚0.00016382897787989208 ' lr-AdamW/pg2‚0.00016382897787989208 trainer/global_step‚5049 % _timestamp‚1.7884321578871806e+09 _runtime‚3842.591902113 ë¬ÎËÈ ' lr-AdamW/pg2‚0.00016382897787989208 trainer/global_step‚5049 % _timestamp‚1.7884321578871806e+09 _runtime‚3842.591902113 _step‚412 ' lr-AdamW/pg1‚0.00016382897787989208œ³%à2Ì- _wandbruntime‚3842  _runtime‚3842^âj7Ý'Í×O D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320901565357568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298112303275520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4232051715072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73511508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚293880907922944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4216469626640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95393556 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67100400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4438725180928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚5049 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚325340098780160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚3842.592806457 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚298112977640960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73511510 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚293880907922944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24207290 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290392056773392 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286175570107392 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168905066 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95373518 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4438557362688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 42320517150721uwXâ'ÎÜO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293880925925888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67099945 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298112303275520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4232051715072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73511508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚293880907922944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4216469626640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95393556 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67100400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4438725180928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚5049 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚325340098780160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚3842.592806457 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚298112977640960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73511510 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚293880907922944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24207290 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290392056773392 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286175570107392 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168905066 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95373518 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4438557362688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4232051715072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320901565357568 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚325402612308992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290447777339124 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚3843.243078834 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚298170150120960 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚320963216095232 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95412036 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚325402797762560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286230474237952 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42901278 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4216631623848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293937260961792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4232214780416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚320963192155136 [MDeviceStatsMonitor.on_train_‡úÂî=batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4439581667328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚5049 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67112694 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286230491277312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67113131 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884321585379326e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168937562 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95412036 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚293937278964736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290447105861800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73525526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42901276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24211418 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4217286061812 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚298170150120960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298169475742208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4439420153856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168917496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168917496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73525524 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚293937278964736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168937562 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2Ëå¶Zì&ÑæM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚325402612308992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290447777339124 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚3843.243078834 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚298170150120960 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚320963216095232 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95412036 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚325402797762560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286230474237952 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42901278 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4216631623848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293937260961792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4232214780416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚320963192155136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4439581667328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚5049 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67112694 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286230491277312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67113131 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884321585379326e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168937562 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95412036 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚293937278964736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290447105861800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73525526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42901276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24211418 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4217286061812 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚298170150120960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298169475742208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4439420153856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168917496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168917496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73525524 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚293937278964736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168937562 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0žKç/^2Ò- _wandbruntime‚3843  _runtime‚3843‘Ub%\ÓÖ 0loss/FRt3/assignment_loss‚3.3270476162433624 2loss/SRqqt2/assignment_loss‚1.6880283329325418 2loss/SRqqt3/assignment_loss‚1.7087609890316213 2loss/SRqqt1/assignment_loss‚1.1478716824721131 1loss/SRqqt3/detection_loss‚2.5742922191108977 trainer/global_step‚5049 0loss/FRt1/assignment_loss‚1.6342032253742218 /loss/FRt1/detection_loss‚1.8117254607217468 /loss/FBt3/detection_loss‚2.6660773243222917 epoch‚4 _step‚415 0loss/FRt2/assignment_loss‚3.3279633841863494 1loss/FBt3/assignment_loss‚0.11761343745248658 .loss/FRt3/detection_loss‚2.692883241507742 .loss/FRt4/detection_loss‚2.363418244502761 %loss/total_loss‚49.91099666959526 /loss/FBt1/assignment_loss‚0.195962479690243 1loss/SRqqt1/detection_loss‚1.0099477706373565 1loss/SRqqt2/detection_loss‚1.8084571128711104 /loss/FBt1/detection_loss‚0.6424720829937872 3loss/FBt2/assignment_loss‚0.0021131346002221107 /loss/FBt4/assignment_loss‚0.141488031251356 .loss/FRt2/detection_loss‚2.294120003537434 1loss/SRqqt4/detection_loss‚1.8781939328841444 /loss/FRt4/assignment_loss‚2.947749911384149 2loss/SRqqt4/assignment_loss‚1.8265317839763637 .loss/FBt2/detection_loss‚8.880571782588959 /loss/FBt4/detection_loss‚3.2235034853219986 % _timestamp‚1.7884321585394082e+09 _runtime‚3843.244047109:¤ÏoaÔÛ % _timestamp‚1.7884321585394082e+09 _runtime‚3843.244047109 0loss/FRt3/assignment_loss‚3.3270476162433624 2loss/SRqqt2/assignment_loss‚1.6880283329325418 2loss/SRqqt3/assignment_loss‚1.7087609890316213 2loss/SRqqt1/assignment_loss‚1.1478716824721131 1loss/SRqqt3/detection_loss‚2.5742922191108977 trainer/global_step‚5049 0loss/FRt1/assignment_loss‚1.6342032253742218 /loss/FRt1/detection_loss‚1.8117254607217468 /loss/FBt3/detection_loss‚2.6660773243222917 epoch‚4 _step‚415 0loss/FRt2/assignment_loss‚3.3279633841863494 1loss/FBt3/assignment_loss‚0.11761343745248658 .loss/FRt3/detection_loss‚2.692883241507742 .loss/FRt4/detection_loss‚2.363418244502761 %loss/total_loss‚49.91099666959526 /loss/FBt1/assignment_loss‚0.195962479690243 1loss/SRqqt1/detection_loss‚1.0099477706373565 1loss/SRqqt2/detection_loss‚1.8084571128711104 /loss/FBt1/detection_loss‚0.6424720829937872 3loss/FBt2/assignment_loss‚0.0021131346002221107 /loss/FBt4/assignment_loss‚0.141488031251356 .loss/FRt2/detection_loss‚2.294120003537434 1loss/SRqqt4/detection_loss‚1.8781939328841444 /loss/FRt4/assignment_loss‚2.947749911384149 2loss/SRqqt4/assignment_loss‚1.8265317839763637 .loss/FBt2/detection_loss‚8.880571782588959 /loss/FBt4/detection_loss‚3.2235034853219986Ÿ¬w×ÍÕ‚(:’ ªžåÔ—„òŽ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.566000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 7ÃÖ‚(: ªžåÔ½¢µ$ disk./.usageGB‚1100.789176940918$ disk.nvme0n1.out‚22464.654296875 disk.rbd1.out‚ 2.08349609375 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚139.9375) proc.memory.availableMB‚ 1424.44140625 disk.rbd2.in‚0 disk.rbd3.out‚0 network.recv‚ 1.443687e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 152.765625 disk.rbd1.in‚0 disk.nvme0c0n1.in‚139.9375 disk.sda2.in‚0 network.sent‚ 1.1268007e+07 disk.sda.in‚0 disk.sda.out‚ 152.765625& disk.nvme0c0n1.out‚22464.654296875 disk.sda2.out‚0! disk.dm-0.out‚22464.654296875$ memory_percent‚91.30589962005615* proc.memory.percent‚17.531394958496094 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.dm-0.in‚139.9375) disk./.usagePercent‚73.87488528814406 disk.rbd3.in‚0 cpu‚25.35620529967403 proc.cpu.threads‚17 disk.rbd0.in‚0! disk.rbd2.out‚121.18408203125 disk.sda3.out‚ 152.765625 disk.sda1.in‚0 proc.memory.rssMB‚ 2872.34375£½~ž×‚(:“ ¹žåԦ斎 gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚160.173' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.034600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òaàØ‚(:‘ ¹žåÔŽ¯À disk.rbd1.out‚ 2.146484375 disk.nvme0c0n1.in‚139.9375 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 153.38671875 disk.dm-0.out‚22563.98046875 disk.sda.out‚ 153.38671875 disk.dm-1.out‚ 153.38671875 disk.nvme0n1.in‚139.9375 disk.rbd3.out‚0 cpu‚25.355025200974694) disk./.usagePercent‚73.87488144803105 disk.sda1.in‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd0.in‚0 proc.memory.rssMB‚ 2872.34375 network.sent‚ 1.1376779e+07 disk.dm-1.in‚0$ disk./.usageGB‚1100.789119720459# disk.nvme0n1.out‚22563.98046875* proc.memory.percent‚17.531394958496094 network.recv‚ 1.450266e+06 disk.rbd1.in‚0% disk.nvme0c0n1.out‚22563.98046875 disk.drbd1777.out‚0( proc.memory.availableMB‚ 1424.4140625 disk.sda1.out‚0! disk.rbd2.out‚121.24658203125 disk.sda3.in‚0 disk.rbd3.in‚0$ memory_percent‚91.30606651306152 disk.dm-0.in‚139.9375 disk.sda.in‚0Aâ¬2Ù- _wandbruntime‚3877  _runtime‚3877ŸÔ|ÉÚà trainer/global_step‚5099 % _timestamp‚1.7884321926781852e+09 _runtime‚3877.382935216 _step‚416 ' lr-AdamW/pg1‚0.00016235954964669087 ' lr-AdamW/pg2‚0.00016235954964669087“c™ÎÛÈ ' lr-AdamW/pg1‚0.00016235954964669087 ' lr-AdamW/pg2‚0.00016235954964669087 trainer/global_step‚5099 % _timestamp‚1.7884321926781852e+09 _runtime‚3877.382935216 _step‚416 ÝšIs2Ü- _wandbruntime‚3877  _runtime‚3877 è]ñÝ'Ý×O YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300971615389184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96299570 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96299570 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300970941023744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚288920776635392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚293177427163684 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4257304966048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚5099 % _timestamp‚1.7884321926788716e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300971615389184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170511878 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43301508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67736676 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚417 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296698577868288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4273037520896 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4272381158400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96319608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24434718 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚328465624600576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67736226 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323984102240768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296698559865344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74212310 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24435166 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4481546299904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038ŒY 5ÞÜO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24435166 _MDeviceStatsMonitor.on_train_batch_start/inactive_ä4Ô!­&split_bytes.small_pool.freed‚ 4481546299904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300971615389184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96299570 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96299570 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300970941023744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚288920776635392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚293177427163684 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4257304966048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚5099 % _timestamp‚1.7884321926788716e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300971615389184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170511878 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43301508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67736676 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚417 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296698577868288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4273037520896 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4272381158400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96319608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24434718 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚328465624600576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67736226 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323984102240768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296698559865344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74212310 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680¡€ðÕ2ß- _wandbruntime‚3878  _runtime‚38784åÎiç&àáM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚301028787869184 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚296754930907136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3878.033856758 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43309678 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296754930907136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4258121401220 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96318024 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96318024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74226326 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170564414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328528151995392 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚293233819206532 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170544348 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67748901 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43309676 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4273200586240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚301028787869184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74226326 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96338088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4257466963256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288975680765952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24439225 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4482422957056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚293233147729208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚5099 _step‚418 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4273856962048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324045729038336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚301028113490432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96338088Ûñèì&áæM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296754930907136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4258121401220 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96318024 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96318024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74226326 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170564414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328528151995392 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚293233819206532 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170544348 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67748901 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43309676 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4273200586240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚301028787869184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74226326 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96338088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4257466963256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288975680765952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24439225 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4482422957056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚293233147729208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚5099 _step‚418 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4273856962048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324045729038336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚301028113490432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96338088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚301028787869184 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚296754930907136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3878.033856758 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43309678 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¢] Ù2â- _wandbruntime‚3878  _runtime‚3878s=â@ZãÔ 2loss/SRqqt1/assignment_loss‚1.0986987361210887 2loss/SRqqt3/assignment_loss‚1.7224615316838026 .loss/FRt2/detection_loss‚2.503275030851364 /loss/FBt2/detection_loss‚3.2549060923712596 /loss/FRt3/assignment_loss‚3.645423947274685 .loss/FRt4/detection_loss‚2.124931359420652 %loss/total_loss‚43.58894509031505 /loss/FRt3/detection_loss‚2.7699085965752603 /loss/FBt3/detection_loss‚2.7762838810682298 trainer/global_step‚5099 _step‚419 0loss/FRt1/assignment_loss‚1.3543857875859961 2loss/SRqqt4/assignment_loss‚1.5445841591406082 /loss/FRt1/detection_loss‚1.6110126010651857 /loss/FBt1/detection_loss‚0.6743163769686052 /loss/FRt2/assignment_loss‚2.405345396697521 0loss/FRt4/assignment_loss‚3.9414717710536458 1loss/FBt1/assignment_loss‚0.20392096314445646 _runtime‚3878.035042372 1loss/FBt3/assignment_loss‚0.04676483268849552 1loss/SRqqt1/detection_loss‚1.2966563378981615 .loss/FBt4/detection_loss‚2.537137973308563 % _timestamp‚1.7884321933303256e+09 1loss/FBt2/assignment_loss‚0.15330417693725654 0loss/FBt4/assignment_loss‚0.3221262860111892 1loss/SRqqt4/detection_loss‚1.9534235750990254 epoch‚4 2loss/SRqqt2/assignment_loss‚1.2798729983243076 /loss/SRqqt2/detection_loss‚1.91247058077292 0loss/SRqqt3/detection_loss‚2.456262098252773#Å5½_äÙ /loss/FRt2/assignment_loss‚2.405345396697521 0loss/FRt4/assignment_loss‚3.9414717710536458 1loss/FBt1/assignment_loss‚0.20392096314445646 _runtime‚3878.035042372 1loss/FBt3/assignment_loss‚0.04676483268849552 1loss/SRqqt1/detection_loss‚1.2966563378981615 .loss/FBt4/detection_loss‚2.537137973308563 % _timestamp‚1.7884321933303256e+09 1loss/FBt2/assignment_loss‚0.15330417693725654 0loss/FBt4/assignment_loss‚0.3221262860111892 1loss/SRqqt4/detection_loss‚1.9534235750990254 epoch‚4 2loss/SRqqt2/assignment_loss‚1.2798729983243076 /loss/SRqqt2/detection_loss‚1.91247058077292 0loss/SRqqt3/detection_loss‚2.456262098252773 2loss/SRqqt1/assignment_loss‚1.0986987361210887 2loss/SRqqt3/assignment_loss‚1.7224615316838026 .loss/FRt2/detection_loss‚2.503275030851364 /loss/FBt2/detection_loss‚3.2549060923712596 /loss/FRt3/assignment_loss‚3.645423947274685 .loss/FRt4/detection_loss‚2.124931359420652 %loss/total_loss‚43.58894509031505 /loss/FRt3/detection_loss‚2.7699085965752603 /loss/FBt3/detection_loss‚2.7762838810682298 trainer/global_step‚5099 _step‚419 0loss/FRt1/assignment_loss‚1.3543857875859961 2loss/SRqqt4/assignment_loss‚1.5445841591406082 /loss/FRt1/detection_loss‚1.6110126010651857 /loss/FBt1/detection_loss‚0.6743163769686052£nOŽ+å‚(:“ ÈžåÔÒÊÚ gpu.0.gpu‚43òU€s.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.166' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.833199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'g+Ûæ‚(:‰ ÈžåÔ¸èë disk.rbd2.in‚0 disk.sda1.out‚0" disk.nvme0n1.out‚ 22564.3203125 disk.rbd1.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚73.87487786392558 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda2.out‚0% disk./.usageGB‚1100.7890663146973 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.dm-0.out‚ 22564.3203125$ memory_percent‚91.30480289459229 disk.sda3.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 153.796875 disk.dm-1.out‚ 153.796875 disk.nvme0n1.in‚139.9375* proc.memory.percent‚17.531394958496094 cpu‚25.353850846142915 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 22564.3203125! disk.rbd2.out‚121.26220703125 disk.dm-1.in‚0) proc.memory.availableMB‚ 1424.62109375 proc.memory.rssMB‚ 2872.34375 proc.cpu.threads‚17 network.sent‚ 1.138228e+07 disk.sda.in‚0 disk.rbd1.out‚ 2.146484375 disk.rbd0.in‚0 network.recv‚ 1.454066e+06 disk.sda3.out‚ 153.796875 disk.dm-0.in‚139.9375 disk.nvme0c0n1.in‚139.9375ñùˆ®’ç‚(:‡ מåÔ’ÃÑ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚92.783' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[*ñ è‚(:• מåÔ‰‹ø disk.dm-1.out‚ 154.34765625% disk.nvme0n1.out‚22663.5576171875 disk.rbd2.out‚121.3486328125' proc.memory.availableMB‚ 949.9921875* proc.memory.percent‚17.531394958496094 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd1.out‚ 2.146484375 disk.nvme0c0n1.in‚139.9375 disk.sda3.out‚ 154.34765625" disk.dm-0.out‚22663.5576171875' disk.nvme0c0n1.out‚22663.5576171875 disk.sda2.out‚0% disk./.usageGB‚1100.7890853881836 disk.rbd3.in‚0 disk.rbd2.in‚0 cpu‚25.35263837123826 network.sent‚ 1.1490076e+07 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0$ memory_percent‚94.20170783996582 disk.dm-0.in‚139.9375 proc.memory.rssMB‚ 2872.34375 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 1.459439e+06 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 154.34765625 disk.nvme0n1.in‚139.9375) disk./.usagePercent‚73.87487914396324 é>32é- _wandbruntime‚3912  _runtime‚3912~¼.Éêà _step‚420 ' lr-AdamW/pg1‚0.00016087989985331808 ' lr-AdamW/pg2‚0.00016087989985331808 trainer/global_step‚5149 % _timestamp‚1.7884322276078784e+09 _runtime‚3912.312598842-z>ÎëÈ ' lr-AdamW/pg2‚0.00016087989985331808 trainer/global_step‚5149 % _timestamp‚1.7884322276078784e+09 _runtime‚3912.312598842 _step‚420 ' lr-AdamW/pg1‚0.00016087989985331808¤DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303829578771968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299516211807744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4297485867700 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303830253137408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97245660 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97245660 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331591417800192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331591226041856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 trainer/global_step‚5149 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299516229810688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4313366964224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327066639123968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291665983163392 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172158770 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24663248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚299516211807744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303830253137408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74913108 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4524610857984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74913110 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4298140305456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚291666000202752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3912.313461568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74913110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68373589 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295964140508208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚303829578771968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4314023326720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43709910 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4524778676224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4314023326720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ÷?á'îÛO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303829578771968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299516211807744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4297485867700 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303830253137408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97245660 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97245660 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331591417800192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331591226041856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 trainer/global_step‚5149 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299516229810688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4313366964224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327066639123968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291665983163392 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172158770 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24663248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚299516211807744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303830253137408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74913108 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4524610857984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74913110 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4298140305456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚291666000202752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3912.313461568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74913110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68373589 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295964140508208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚303829578771968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4314023326720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43709910 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4524778676224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4314023326720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚421¥H ¦2ï- _wandbruntime‚3912  _runtime‚3912Ÿçð•‰"ðáM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4298302302664 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299572582849536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4314842767872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4525635899904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4314186392064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97244076 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4314186392064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24668213 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327128289861632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331653925761536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327128265921536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303887425617408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172191266 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172171200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291720904333312 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97244076 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303887425617408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 trainer/global_step‚5149 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884322282666173e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296019861073940 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74927126 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303886751238656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299572564846592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97264140 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4314842767872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4298956740628 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97264140 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68386291 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4525470192128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24667771 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68385847 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74927124 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43718078 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172191266 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚299572582849536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299572564846592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74927126 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _step‚422 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4Zz¶ü^3718076 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331653736113664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚3912.971903857 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299572582849536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4314842767872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4525635899904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4314186392064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97244076 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4314186392064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24668213 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327128289861632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331653925761536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327128265921536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303887425617408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172191266 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172171200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291720904333312 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97244076 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303887425617408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 trainer/global_step‚5149 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884322282666173e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296019861073940 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74927126 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303886751238656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299572564846592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97264140 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4314842767872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4298956740628 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97264140 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68386291 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4525470192128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24667771 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68385847 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74927124 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43718078 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172191266 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚299572582849536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299572564846592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74927126 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _step‚422 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43718076 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331653736113664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚3912.971903857 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75613910 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334717091783168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4354352770048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302333881753088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44118310 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44118308 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294411206730752 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75613910 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306688890885632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98151674 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24892651 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98171712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306688216520192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302333881753088 trainer/global_step‚5199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4567915776000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884322624328353e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306688216520192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330149152067072 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24892212 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302333863750144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69010520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚298749510898500 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚3947.137788214 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173765582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚298750182375616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75613908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338975644864 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928—¿÷9d€ÜO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24892212 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302333863750144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69010520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚298749510898500 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚3947.137788214 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173765582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚298750182375616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75613908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338975644864 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4355009132544 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75613910 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334717091783168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4354352770048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302333881753088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44118310 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44118308 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294411206730752 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75613910 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306688890885632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98151674 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24892651 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98171712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306688216520192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302333881753088 trainer/global_step‚5199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4567915776000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884322624328353e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306688216520192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330149152067072 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430©ÇĶ2- _wandbruntime‚3948  _runtime‚3948™Û²*ç&‚áM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75627926 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334779598444032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298805231464024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294466093821952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173818118 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98170128 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302390234791936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75627924 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330210826744832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4339792080036 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4355172197888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44126476 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚426 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98190192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75627924 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884322633142536e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306746063365632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302390216788992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚334779408796160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4355828573696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173798052 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚5199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302390216788992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98190192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44126478 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302390234791936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306746063365632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98170128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69023570 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173798052 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330210802804736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚3948.019542727 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173818118 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75627926 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69023125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24896649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294466110861312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306745388986880 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4355172197888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24897092 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298805902941348 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339137642072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4568605991424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4355828573696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445¾5~ì&ƒæM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306746063365632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98170128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69023570 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173798052 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330210802804736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚3948.019542727 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173818118 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75627926 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69023125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24896649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294466110861312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306745388986880 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4355172197888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24897092 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298805902941348 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339137642072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4568605991424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4355828573696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75627926 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334779598444032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298805231464024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294466093821952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173818118 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98170128 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302390234791936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75627924 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330210826744832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4339792080036 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4355172197888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44126476 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚426 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98190192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75627924 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884322633142536e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306746063365632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302390216788992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚334779408796160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4355828573696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173798052 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚5199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302390216788992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98190192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44126478 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302390234791936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680ª[oi2„- _wandbruntime‚3948  _runtime‚3948hw‹n]…× 1loss/FBt3/assignment_loss‚0.01924017971177818 3loss/FBt4/assignment_loss‚0.0040423704078420995 _step‚427 0loss/FRt1/assignment_loss‚1.1376671195030212 -loss/FRt3/detection_loss‚2.68886880616884 /loss/FRt4/detection_loss‚2.1091276245602106 /loss/FBt3/detection_loss‚2.5686076462268828 epoch‚4 0loss/FRt4/assignment_loss‚3.2090699026140115 0loss/SRqqt3/assignment_loss‚1.93853578902781 /loss/FRt2/detection_loss‚2.5031783000016823 -loss/FBt2/detection_loss‚3.30413771337933 % _timestamp‚1.7884322633159692e+09 /loss/FRt2/assignment_loss‚3.130963753431271 2loss/SRqqt4/assignment_loss‚1.4344200427591238 1loss/SRqqt1/detection_loss‚1.0628329371347238 0loss/SRqqt3/detection_loss‚2.141330884261565 /loss/FRt1/detection_loss‚1.7435684840126735 /loss/FBt1/detection_loss‚0.5854125864327685 _runtime‚3948.020591283 2loss/SRqqt1/assignment_loss‚0.9089740504706768 1loss/SRqqt2/assignment_loss‚1.846040989127424 2loss/FBt2/assignment_loss‚0.019689124818088526 1loss/SRqqt4/detection_loss‚1.7364547694066785 %loss/total_loss‚42.62114652691511 trainer/global_step‚5199 0loss/FRt3/assignment_loss‚3.4293298173595117 1loss/FBt1/assignment_loss‚0.12875219220713968 1loss/SRqqt2/detection_loss‚1.8952675604157978 /loss/FBt4/detection_loss‚3.0756338834762573õ>}b†Ü 1loss/FBt3/assignment_loss‚0.01924017971177818 3loss/FBt4/assignment_loss‚0.0040423704078420995 _step‚427 0loss/FRt1/assignment_loss‚1.1376671195030212 -loss/FRt3/detection_loss‚2.68886880616884 /loss/FRt4/detection_loss‚2.1091276245602106 /loss/FBt3/detection_loss‚2.5686076462268828 epoch‚4 0loss/FRt4/assignment_loss‚3.2090699026140115 0loss/SRqqt3/assignment_loss‚1.93853578902781 /loss/FRt2/detection_loss‚2.5031783000016823 -loss/FBt2/detection_loss‚3.30413771337933 % _timestamp‚1.7884322633159692e+09 /loss/FRt2/assignment_loss‚3.130963753431271 2loss/SRqqt4/assignment_loss‚1.4344200427591238 1loss/SRqqt1/detection_loss‚1.0628329371347238 0loss/SRqqt3/detection_loss‚2.141330884261565 /loss/FRt1/detection_loss‚1.7435684840126735 /loss/FBt1/detection_loss‚0.5854125864327685 _runtime‚3948.020591283 2loss/SRqqt1/assignment_loss‚0.9089740504706768 1loss/SRqqt2/assignment_loss‚1.846040989127424 2loss/FBt2/assignment_loss‚0.019689124818088526 1loss/SRqqt4/detection_loss‚1.7364547694066785 %loss/total_loss‚42.62114652691511 trainer/global_step‚5199 0loss/FRt3/assignment_loss‚3.4293298173595117 1loss/FBt1/assignment_loss‚0.12875219220713968 1loss/SRqqt2/detection_loss‚1.8952675604157978 /loss/FBt4/detection_loss‚3.0756338834762573«Úw¢¸ž‡‚(:“ “ŸåÔ𳂎 gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.304' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.860799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ybjŠ#ˆ‚(:˜ “ŸåÔñ‰³ disk.sda.out‚ 156.5234375) disk./.usagePercent‚73.87488682418926 disk.nvme0n1.in‚139.9375 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚0 proc.memory.rssMB‚ 2872.34375 proc.cpu.threads‚17& disk.nvme0n1.out‚22962.04248046875 disk.sda1.in‚0( proc.memory.availableMB‚ 949.98046875* proc.memory.percent‚17.531394958496094 cpu‚25.347951239846587( disk.nvme0c0n1.out‚22962.04248046875 disk.rbd0.out‚0 disk.sda3.in‚0$ memory_percent‚94.20177936553955 disk.rbd1.out‚ 2.146484375 disk.sda3.out‚ 156.5234375# disk.dm-0.out‚22962.04248046875 disk.sda1.out‚0 disk.dm-1.out‚ 156.5234375 network.sent‚ 1.1615907e+07 disk.nvme0c0n1.in‚139.9375 disk.rbd3.in‚0 network.recv‚ 1.477944e+06% disk./.usageGB‚1100.7891998291016 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0! disk.rbd2.out‚123.04052734375 disk.drbd1777.out‚0 disk.dm-0.in‚139.9375 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda.in‚0Ø…ìÁ“‰‚(:ˆ ¢ŸåÔ¦ñê gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.842' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…㶚$Š‚(:™ ¢ŸåÔÍèý disk.rbd1.out‚ 2.146484375! disk.rbd2.out‚123.05615234375 network.sent‚ 1.1723763e+07) disk./.usagePercent‚73.87489015228721 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.nvme0n1.in‚139.9375 disk.rbd1.in‚0 disk.rbd3.out‚0 proc.memory.rssMB‚ 2872.34375 disk.drbd1777.in‚0 disk.dm-0.in‚139.9375# disk.dm-0.out‚23061.13623046875 disk.sda1.in‚0 disk.nvme0c0n1.in‚139.9375( disk.nvme0c0n1.out‚23061.13623046875 disk.rbd2.in‚0$ disk./.usageGB‚1100.789249420166& disk.nvme0n1.out‚23061.13623046875 disk.drbd1777.out‚0$ memory_percent‚94.20180320739746* proc.memory.percent‚17.531394958496094 cpu‚25.346902207932068 disk.sda.out‚ 156.83984375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 156.83984375' proc.memory.availableMB‚ 949.9765625 disk.sda3.out‚ 156.83984375 network.recv‚ 1.483645e+06 disk.rbd0.in‚0_£œþ2‹- _wandbruntime‚3982  _runtime‚3982lõ Ɍà ' lr-AdamW/pg1‚0.00015789117107915143 ' lr-AdamW/pg2‚0.00015789117107915143 trainer/global_step‚5249 % _timestamp‚1.7884322975574186e+09 _runtime‚3982.262175042 _step‚428l#Å·È ' lr-AdamW/pg1‚0.00015789117107915143 ' lr-AdamW/pg2‚0.00015789117107915143 trainer/global_step‚5249 % _timestamp‚1.7884322975574186e+09 _runtime‚3982.262175h¬ü042 _step‚428¬Ã:7!2Ž- _wandbruntime‚3982  _runtime‚3982È3âÜ'ÖO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175392434 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301535552765908 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25120920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚309546854268416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69647628 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99097764 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4610890954752 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305151533695488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4395994938368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44526708 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4395338575872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4379156546516 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301536224243024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 _runtime‚3982.263107769 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4395338575872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175412474 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76314710 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4611050384384 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 $ _timestamp‚1.788432297558159e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333231712890368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44526710 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚337842579905024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚5249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305151515692544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297156396219392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309546854268416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69648079 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337842763274752 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚309547528633856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99077726 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25121369 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305151533695488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333231688950272 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76314710 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309547528633856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚429 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333231712890368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44526710 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚337842579905024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚5249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305151515692544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297156396219392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309546854268416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69648079 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337842763274752 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚309547528633856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99077726 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25121369 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305151533695488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333231688950272 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76314710 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309547528633856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚429 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76328726 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333293363628032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚309604701113856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25125873 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚297211317389312 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99116244 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175444970 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44534876 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884322982105498e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309604701113856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69660296 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175424904 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175424904 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69660751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3982.915945521 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76328724 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4379972981480 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚305207868731392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305207886734336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4611915013632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76328726 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309604026735104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4396158003712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337905278641664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301591273331432 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305207868731392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175444970 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414T2J[°“æM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337905091090944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚5249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚305207886734336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4396814379520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99096180 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175444970 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4396814379520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25125420 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚I†ã< 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76328726 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333293363628032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚309604701113856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25125873 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚297211317389312 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99116244 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175444970 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44534876 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884322982105498e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309604701113856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69660296 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175424904 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175424904 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69660751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3982.915945521 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76328724 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4379972981480 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚305207868731392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305207886734336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4611915013632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76328726 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309604026735104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4396158003712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337905278641664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301591273331432 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305207868731392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312405492016640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77015510 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299901619786752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307969167634944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4017.427032246 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304322266110432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚312405492016640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100023816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340968315846144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177039326 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77015508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25349551 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304321594633316 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25349117 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4654066072576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4436324381696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70284661 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚433 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4419991885924 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚312406166382080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44935108 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77015510 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4436980744192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4653906642944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100023816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952†µDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312405492016640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77015510 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299901619786752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307969167634944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4017.427032246 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304322266110432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚312405492016640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100023816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340968315846144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177039326 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77015508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25349551 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304321594633316 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMoni¾ðùªtor.on_train_batch_start/segment.all.allocated‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25349117 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4654066072576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4436324381696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70284661 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0±D‰Jæ2¡- _wandbruntime‚4018  _runtime‚4018*°Ìë&¢åM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44943278 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4654751835648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100022232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77029524 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚304377986676164 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚434 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚308025520673792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77029524 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437800185344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177071822 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70296932 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4654921737728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299956523917312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77029526 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312463338862080 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100042296 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304377315198840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308025538676736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25354106 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312463338862080 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70297384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 _runtime‚4018.087354079 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4437143809536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177051756 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚5299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341030628406784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308025538676736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336375876571136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4437800185344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341030822248960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚299956506877952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44943276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25353656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177051756 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100042296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100022232 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100042296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100022232 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚304377986676164 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚434 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚308025520673792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77029524 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437800185344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177071822 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70296932 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4654921737728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299956523917312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77029526 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312463338862080 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100042296 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304377315198840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308025538676736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25354106 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312463338862080 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70297384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 _runtime‚4018.087354079 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4437143809536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177051756 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚5299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341030628406784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308025538676736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336375876571136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4437800185344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341030822248960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚299956506877952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44943276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25353656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177051756 PnÉ ±ÛO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315264129764864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4477310187520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚315264804130304 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4461481663088 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77716308 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339396786656768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4697068612096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4460827225332 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100929830 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚315264804130304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100949868 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178646138 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4477966550016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307108307977840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚4052.310822184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70921188 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344093831328768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25577680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310786837580288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77716310 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100929830 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚307107636500724 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302646809275392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310786819577344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178666178 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5349 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4477966550016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45343510 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4697223847424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚425 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302646826314752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0F3g~æ'²àO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚315264129764864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310786819577344 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77716308 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339396786656768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4697068612096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4460827225332 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100929830 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚315264804130304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100949868 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178646138 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4477966550016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307108307977840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚4052.310822184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70921188 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344093831328768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25577680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310786837580288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77716310 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100929830 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚307107636500724 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302646809275392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310786819577344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178666178 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5349 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4477966550016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45343510 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4697223847424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚425 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302646826314752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315264129764864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4477310187520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚315264804130304 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4461481663088 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307164028543572 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77730326 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚307163357066248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310843172616192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178698674 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77730326 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70934275 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25582597 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310843190619136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4478129615360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45351678 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339458437394432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚315321976610304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚315321302231552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178678608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45351676 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339458413454336 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚429 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100968348 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344156523955200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4698086560768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4697918755840 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚438 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178698674 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70933846 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚5349 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4478785991168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25582170 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344156332210176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚4052.965528881 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚315321976610304 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178678608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77730324 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 % _timestamp‚1.7884323682600994e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4461643660296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100968348 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4478129615360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302701713405952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310843190619136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100948284 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100948284„¼ µêM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307164028543572 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77730326 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚307163357066248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310843172616192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178698674 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77730326 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70934275 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25582597 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310843190619136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4478129615360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45351678 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339458437394432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚315321976610304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚315321302231552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178678608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45351676 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339458413454336 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚429 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100968348 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344156523955200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4698086560768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4697918755840 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚438 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178698674 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70933846 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚5349 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4478785991168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25582170 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344156332210176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚4052.965528881 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚315321976610304 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_env¿^.d/allocation.all.freed‚ 178678608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77730324 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 % _timestamp‚1.7884323682600994e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4461643660296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100968348 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4478129615360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302701713405952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310843190619136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100948284 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100948284 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4478785991168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1¶]Ê¥2¶- _wandbruntime‚4052  _runtime‚4052×]í]·× 0loss/SRqqt3/detection_loss‚2.261418387719563 trainer/global_step‚5349 0loss/FRt1/assignment_loss‚1.4808813286762612 /loss/FRt2/assignment_loss‚2.969792960620508 0loss/FRt4/assignment_loss‚3.1416317224502563 2loss/SRqqt1/assignment_loss‚0.8067591880338595 0loss/FBt3/assignment_loss‚0.1731995542063487 % _timestamp‚1.7884323682619035e+09 0loss/FRt3/assignment_loss‚3.2529611669737717 0loss/FBt1/assignment_loss‚0.0954961002114894 1loss/SRqqt1/detection_loss‚1.0834898455616306 .loss/FBt2/detection_loss‚3.320994019508362 _step‚439 1loss/SRqqt2/detection_loss‚2.0723410894473395 epoch‚4 /loss/FRt3/detection_loss‚2.9623993080237816 _runtime‚4052.966867252 2loss/SRqqt3/assignment_loss‚2.2118399385895047 0loss/SRqqt4/assignment_loss‚1.54666020582288 /loss/FRt4/detection_loss‚2.3128522388063946 0loss/SRqqt4/detection_loss‚1.510565758761713 2loss/FBt2/assignment_loss‚0.017002531202160755 /loss/FBt1/detection_loss‚0.6355774123165314 /loss/FRt2/detection_loss‚2.4764569518042774 /loss/FBt3/detection_loss‚2.6002104708126614 /loss/FBt4/detection_loss‚1.8704280734062195 &loss/total_loss‚41.874600659040496 2loss/SRqqt2/assignment_loss‚1.2865431266171592 1loss/FBt4/assignment_loss‚0.06360892038792372 /loss/FRt1/detection_loss‚1.7214903590798962x2SÇb¸Ü 1loss/SRqqt2/detection_loss‚2.0723410894473395 epoch‚4 /loss/FRt3/detection_loss‚2.9623993080237816 _runtime‚4052.966867252 2loss/SRqqt3/assignment_loss‚2.2118399385895047 0loss/SRqqt4/assignment_loss‚1.54666020582288 /loss/FRt4/detection_loss‚2.3128522388063946 0loss/SRqqt4/detection_loss‚1.510565758761713 2loss/FBt2/assignment_loss‚0.017002531202160755 /loss/FBt1/detection_loss‚0.6355774123165314 /loss/FRt2/detection_loss‚2.4764569518042774 /loss/FBt3/detection_loss‚2.6002104708126614 /loss/FBt4/detection_loss‚1.8704280734062195 &loss/total_loss‚41.874600659040496 2loss/SRqqt2/assignment_loss‚1.2865431266171592 1loss/FBt4/assignment_loss‚0.06360892038792372 /loss/FRt1/detection_loss‚1.7214903590798962 0loss/SRqqt3/detection_loss‚2.261418387719563 trainer/global_step‚5349 0loss/FRt1/assignment_loss‚1.4808813286762612 /loss/FRt2/assignment_loss‚2.969792960620508 0loss/FRt4/assignment_loss‚3.1416317224502563 2loss/SRqqt1/assignment_loss‚0.8067591880338595 0loss/FBt3/assignment_loss‚0.1731995542063487 % _timestamp‚1.7884323682619035e+09 0loss/FRt3/assignment_loss‚3.2529611669737717 0loss/FBt1/assignment_loss‚0.0954961002114894 1loss/SRqqt1/detection_loss‚1.0834898455616306 .loss/FBt2/detection_loss‚3.320994019508362 _step‚439·µx瓹‚(:ˆ üŸåÔ¶Ðñ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.039' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`D$mº‚(: üŸåÔѱà‘ disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 1.1964086e+07 disk.rbd0.in‚0 disk.rbd2.out‚ 124.947265625 disk.dm-0.out‚23557.80859375 disk.sda.out‚ 160.46875 proc.memory.rssMB‚ 2872.34375( disk./.usagePercent‚73.8749999795194" disk.nvme0c0n1.in‚ 409.17578125 disk.sda1.in‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.531394958496094 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 409.17578125 disk.dm-1.in‚0# disk.nvme0n1.out‚23557.80859375$ memory_percent‚94.20280456542969$ disk./.usageGB‚1100.790885925293 disk.rbd1.in‚0 disk.rbd2.in‚0 cpu‚25.340225556492875 network.recv‚ 1.513183e+06 disk.dm-1.out‚ 160.46875$ proc.memory.availableMB‚949.8125 disk.rbd1.out‚ 2.146484375% disk.nvme0c0n1.out‚23557.80859375 disk.rbd3.in‚0 disk.nvme0n1.in‚ 409.17578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 160.46875 disk.sda.in‚0¸ëßv“»‚(:ˆ ‹ åÔ¢Œñ gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<£Ù|¼‚(:“ ‹ åÔð㟠disk.sda.in‚0 disk.rbd0.in‚0 disk.sda.out‚161.1875 network.recv‚ 1.519702e+06 disk.dm-1.out‚161.1875" disk.nvme0c0n1.in‚ 409.17578125 disk.sda2.out‚0 network.sent‚ 1.2072937e+07# disk.dm-0.out‚23657.02880859375 disk.sda1.in‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 2872.34375( disk./.usagePercent‚73.8749999795194 disk.dm-0.in‚ 409.17578125 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.out‚161.1875 disk.rbd3.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚1100.790885925293 disk.sda1.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 409.17578125& disk.nvme0n1.out‚23657.02880859375( disk.nvme0c0n1.out‚23657.02880859375 disk.rbd2.out‚ 125.072265625 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0$ memory_percent‚94.20280456542969* proc.memory.percent‚17.531394958496094 cpu‚25.339123637877265 disk.rbd1.out‚ 2.146484375 disk.sda3.in‚0$ proc.memory.availableMB‚949.8125¹Ò+2½- _wandbruntime‚4087  _runtime‚4087‹‰fáɾà ' lr-AdamW/pg1‚0.00015333919091680822 ' lr-AdamW/pg2‚0.00015333919091680822 trainer/global_step‚5399 % _timestamp‚1.7884324025495665e+09 _runtime‚4087.254430777 _step‚440óD‰!Î¿È ' lr-AdamW/pg1‚0.00015333919091680822 ' lr-AdamW/pg2‚0.00015333919091680822 trainer/global_step‚5399 % _timestamp‚1.7884324025495665e+09 _runtime‚4087.254430777 _step‚440¸Ê?º 2À- _wandbruntime‚4087  _runtime‚4087Û´Mà'ÁÚO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚342479323539968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4518952355840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313604471519744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342479299599872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4740105114624 $ _timestamp‚1.788432402550413e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313604489522688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4518952355840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309893678368132 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71558870 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45751908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25806960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318122767513088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4740262447104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305392032842752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318122767513088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4518295993344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45751910 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71558445 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78417110 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101875920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309894349845248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚4087.255381048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚318123441878528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101855882 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚5399 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25806537 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4502317002496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347219585987072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313604489522688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305392015803392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318123441878528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4518295993344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚5399 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25806537 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4502317002496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347219585987072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313604489522688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305392015803392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318123441878528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4518295993344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71558870 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45751908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25806960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318122767513088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4740262447104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305392032842752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318122767513088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4518295993344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45751910 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71558445 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78417110 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101875920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309894349845248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚4087.255381048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680¹¾H#A2Ã- _wandbruntime‚4087  _runtime‚4087cA}û#ÄåM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚318180614358528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25811541 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4519115421184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4503133437668 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚5399 _runtime‚4087.913128726 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101874336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78431124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45760076 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884324032076795e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313660824558592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4519771796992 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342540974277632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚442 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4519771796992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313660824558592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342540950337536 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78431126 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318180614358528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4519115421184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313660842561536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313660842561536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180305460 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101874336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25811116 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318179939979776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101894400 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101894400 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4741128696320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180325526 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4502478999704 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78431124 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71571192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305446919933952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180305460 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71571619 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78431126 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872«}¿ð&ÅêM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78431126 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚318180614358528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25811541 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4519115421184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4503133437668 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚5399 _runtime‚4087.913128726 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101874336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78431124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45760076 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884324032076795e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313660824558592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4519771796992 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342540974277632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚442 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4519771796992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313660824558592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342540950337536 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78431126 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318180614358528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4519115421184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313660842561536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313660842561536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180305460 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101874336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25811116 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318179939979776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101894400 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101894400 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4741128696320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180325526 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4502478999704 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78431124 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71571192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305446919933952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180305460 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71571619º"шž2Æ- _wandbruntime‚4087  _runtime‚4087Ò´íÎZÇÔ -loss/FBt2/detection_loss‚3.04168630639712 /loss/FRt4/assignment_loss‚3.032948033229725 0loss/SRqqt2/assignment_loss‚1.42162052768728 1loss/SRqqt1/detection_loss‚0.8618062331740345 1loss/SRqqt3/detection_loss‚2.2280822521761845 1loss/SRqqt4/detection_loss‚1.9587398260197741 /loss/FBt3/detection_loss‚2.9896960769380843 .loss/FBt4/detection_loss‚2.418604188495212 .loss/FRt2/detection_loss‚2.311267848108329 /loss/FRt3/detection_loss‚2.6830932372494747 % _timestamp‚1.7884324032094517e+09 1loss/FBt4/assignment_loss‚0.04565038666543034 .loss/FRt1/detection_loss‚1.629363971999136 /loss/FBt1/detection_loss‚0.7692244762820762 %loss/total_loss‚41.46241288109043 _step‚443 /loss/FRt1/assignment_loss‚1.353028002431837 0loss/FRt3/assignment_loss‚2.9558383304821816 2loss/SRqqt3/assignment_loss‚1.5847286338868893 epoch‚4 trainer/global_step‚5399 2loss/SRqqt4/assignment_loss‚1.6817697703521302 _runtime‚4087.914125103 1loss/FBt1/assignment_loss‚0.09398571046468239 3loss/FBt2/assignment_loss‚0.0010050283881355426 2loss/FBt3/assignment_loss‚0.003200389020743647 1loss/SRqqt2/detection_loss‚1.9752066161321558 /loss/FRt2/assignment_loss‚2.803958212628084 1loss/SRqqt1/assignment_loss‚0.886039676251156 .loss/FRt4/detection_loss‚2.731869146630571‡¸_ÈÙ /loss/FRt3/detection_loss‚2.6830932372494747 % _timestamp‚1.7884324032094517e+09 1loss/FBt4/assignment_loss‚0.04565038666543034 .loss/FRt1/detection_loss‚1.629363971999136 /loss/FBt1/detection_loss‚0.7692244762820762 %loss/total_loss‚41.46241288109043 _step‚443 /loss/FRt1/assignment_loss‚1.353028002431837 0loss/FRt3/assignment_loss‚2.9558383304821816 2loss/SRqqt3/assignment_loss‚1.5847286338868893 epoch‚4 trainer/global_step‚5399 2loss/SRqqt4/assignment_loss‚1.6817697703521302 _runtime‚4087.914125103 1loss/FBt1/assignment_loss‚0.09398571046468239 3loss/FBt2/assignment_loss‚0.0010050283881355426 2loss/FBt3/assignment_loss‚0.003200389020743647 1loss/SRqqt2/detection_loss‚1.9752066161321558 /loss/FRt2/assignment_loss‚2.803958212628084 1loss/SRqqt1/assignment_loss‚0.886039676251156 .loss/FRt4/detection_loss‚2.731869146630571 -loss/FBt2/detection_loss‚3.04168630639712 /loss/FRt4/assignment_loss‚3.032948033229725 0loss/SRqqt2/assignment_loss‚1.42162052768728 1loss/SRqqt1/detection_loss‚0.8618062331740345 1loss/SRqqt3/detection_loss‚2.2280822521761845 1loss/SRqqt4/detection_loss‚1.9587398260197741 /loss/FBt3/detection_loss‚2.9896960769380843 .loss/FBt4/detection_loss‚2.418604188495212 .loss/FRt2/detection_loss‚2.311267848108329»›#j^‘É‚(:† š åÔ¶šŒŽ gpu.0.gpu‚59.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.12' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.624 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iOhì'Ê‚(:œ š åÔÙ¿½‘ disk.sda1.in‚0& proc.memory.availableMB‚ 949.578125* proc.memory.percent‚17.531394958496094 network.sent‚ 1.2078112e+07 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 409.17578125 proc.memory.rssMB‚ 2872.34375( disk.nvme0c0n1.out‚23657.39111328125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 409.17578125 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 161.6796875 disk.rbd2.in‚0 disk.dm-1.out‚ 161.6796875& disk.nvme0n1.out‚23657.39111328125# memory_percent‚94.2042350769043 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 126.599609375 disk.sda1.out‚0 disk.dm-1.in‚0 cpu‚25.33805567444335 disk.rbd1.out‚ 2.146484375 network.recv‚ 1.523465e+06( disk./.usagePercent‚73.8749999795194$ disk./.usageGB‚1100.790885925293 disk.sda3.in‚0 disk.sda3.out‚ 161.6796875 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 409.17578125 disk.drbd1777.in‚0# disk.dm-0.out‚23657.39111328125 disk.sda.in‚0Þ BQ‘Ë‚(:† © åÔ’äõ gpu.0.gpu‚6.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚92.434' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±®Å].Ì‚(:£ © åÔóâ” network.sent‚ 1.218694e+07" disk.nvme0c0n1.in‚ 409.17578125 disk.rbd0.in‚0' proc.memory.availableMB‚ 949.4296875 disk.sda2.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 162.14453125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 2.146484375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 cpu‚25.336967237086448 disk.sda.in‚0( disk.nvme0c0n1.out‚23756.84423828125 disk.dm-0.in‚ 409.17578125 disk.sda.out‚ 162.14453125 disk.nvme0n1.in‚ 409.17578125& disk.nvme0n1.out‚23756.84423828125 disk.dm-1.out‚ 162.14453125% disk./.usageGB‚1100.7910118103027 disk.sda3.in‚0 disk.drbd1777.out‚0$ memory_percent‚94.20514106750488 disk.dm-1.in‚0 disk.rbd2.out‚ 126.724609375 network.recv‚ 1.529849e+06) disk./.usagePercent‚73.87500842776802 disk.rbd0.out‚0 disk.drbd1777.in‚0* proc.memory.percent‚17.531394958496094# disk.dm-0.out‚23756.84423828125 proc.memory.rssMB‚ 2872.34375®%2Í- _wandbruntime‚4122  _runtime‚41221kåqÇÎÁ & lr-AdamW/pg1‚0.0001518047572887047 & lr-AdamW/pg2‚0.0001518047572887047 trainer/global_step‚5449 % _timestamp‚1.7884324373914068e+09 _runtime‚4122.096129033 _step‚444Rð¬ZÌÏÆ & lr-AdamW/pg1‚0.0001518047572887047 & lr-AdamW/pg2‚0.0001518047572887047 trainer/global_step‚5449 % _timestamp‚1.7884324373914068e+09 _runtime‚4122.096129033 _step‚444¼Q)Öå2Ð- _wandbruntime‚4122  _runtime‚4122ÿÿÑÛO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46160308 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308137239370752 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312680391712656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4543152341904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4122.097004573 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320982079626752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚445 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4559281799168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79117908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320981405261312 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72196268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345561836483072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312679720235540 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4542497904148 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5449 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79117910 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4559938161664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316422141465088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26035506 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316422123462144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79117910 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350345393842688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884324373921077e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46160310 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316422141465088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320981405261312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308137222331392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350345199987200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320982079626752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181919882 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316422123462144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128v`bÍæ'ÒàO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79117908 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46160308 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308137239370752 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312680391712656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4543152341904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4122.097004573 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320982079626752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚445 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4559281799168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79117908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320981405261312 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72196268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345561836483072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312679720235540 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4542497904148 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5449 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79117910 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4559938161664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316422141465088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26035506 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316422123462144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79117910 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350345393842688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884324373921077e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46160310 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316422141465088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320981405261312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308137222331392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350345199987200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320982079626752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181919882 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316422123462144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46168478 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚446 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4560757602816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4560101227008 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79131924 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321039252106752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316478494503936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79131926 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102820452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321038577728000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4560101227008 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308192126461952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102820452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 trainer/global_step‚5449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46168476 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72209073 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181952378 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4560757602816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102800388 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308192143501312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102800388 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350407892956672 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4784211893760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4122.753766244 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321039252106752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4543968777076 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181952378 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4784381795840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316478494503936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26040133 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72208609 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414ŠÈ:­ð&ÕêM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4560757602816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4560101227008 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79131924 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321039252106752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316478494503936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79131926 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102820452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321038577728000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4560101227008 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308192126461952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102820452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 trainer/global_step‚5449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46168476 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72209073 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181952378 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4560757602816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102800388 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308192143501312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102800388 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350407892956672 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4784211893760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4122.753766244 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321039252106752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4543968777076 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181952378 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4784381795840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316478494503936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26040133 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72208609 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316478476500992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46168478 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚446¾¹Çäp2Ö- _wandbruntime‚4122  _runtime‚412276¡eV×Õ 0loss/FRt4/assignment_loss‚2.9010500198318843 0loss/FBt3/assignment_loss‚0.4461862253180828 1loss/SRqqt4/detection_loss‚1.8033931072877378 /loss/FBt4/detection_loss‚2.1417022293264214 epoch‚4 0loss/FRt3/assignment_loss‚4.0410971677664556 1loss/SRqqt4/assignment_loss‚1.352433375177943 0loss/SRqqt3/detectUǾ‘ion_loss‚2.256467914581299 trainer/global_step‚5449 % _timestamp‚1.7884324380500474e+09 _runtime‚ 4122.75465578 0loss/FRt2/assignment_loss‚2.9748867133916437 1loss/FBt1/assignment_loss‚0.12412748803466161 /loss/FRt1/detection_loss‚1.5776737895276811 /loss/FBt2/detection_loss‚3.0481143508638655 2loss/SRqqt1/assignment_loss‚0.8895272513354818 0loss/FRt1/assignment_loss‚1.2531658256495442 .loss/FBt4/assignment_loss‚0.60588007876967 /loss/FBt1/detection_loss‚0.7097333347068088 /loss/FBt3/detection_loss‚3.1793637360845293 _step‚447 2loss/SRqqt3/assignment_loss‚1.6919676301734787 1loss/FBt2/assignment_loss‚0.47400438431733555 .loss/FRt2/detection_loss‚2.212162524461746 1loss/SRqqt1/detection_loss‚0.8794223714371522 /loss/FRt3/detection_loss‚2.9072257497093896 1loss/SRqqt2/detection_loss‚1.6039102703332901 2loss/SRqqt2/assignment_loss‚1.6773016927142939 /loss/FRt4/detection_loss‚2.2622839382716586 %loss/total_loss‚43.01308116907205Oª"÷`ØÚ .loss/FBt4/assignment_loss‚0.60588007876967 /loss/FBt1/detection_loss‚0.7097333347068088 /loss/FBt3/detection_loss‚3.1793637360845293 _step‚447 2loss/SRqqt3/assignment_loss‚1.6919676301734787 1loss/FBt2/assignment_loss‚0.47400438431733555 .loss/FRt2/detection_loss‚2.212162524461746 1loss/SRqqt1/detection_loss‚0.8794223714371522 /loss/FRt3/detection_loss‚2.9072257497093896 1loss/SRqqt2/detection_loss‚1.6039102703332901 2loss/SRqqt2/assignment_loss‚1.6773016927142939 /loss/FRt4/detection_loss‚2.2622839382716586 %loss/total_loss‚43.01308116907205 0loss/FRt4/assignment_loss‚2.9010500198318843 0loss/FBt3/assignment_loss‚0.4461862253180828 1loss/SRqqt4/detection_loss‚1.8033931072877378 /loss/FBt4/detection_loss‚2.1417022293264214 epoch‚4 0loss/FRt3/assignment_loss‚4.0410971677664556 1loss/SRqqt4/assignment_loss‚1.352433375177943 0loss/SRqqt3/detection_loss‚2.256467914581299 trainer/global_step‚5449 % _timestamp‚1.7884324380500474e+09 _runtime‚ 4122.75465578 0loss/FRt2/assignment_loss‚2.9748867133916437 1loss/FBt1/assignment_loss‚0.12412748803466161 /loss/FRt1/detection_loss‚1.5776737895276811 /loss/FBt2/detection_loss‚3.0481143508638655 2loss/SRqqt1/assignment_loss‚0.8895272513354818 0loss/FRt1/assignment_loss‚1.2531658256495442¿pëÌ(Ù‚(:’ ¸ åÔíÍê gpu.0.gpu‚44.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.274' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.454800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þg-Ú‚(:¢ ¸ åԤƒ& disk.nvme0n1.out‚23855.94677734375' proc.memory.availableMB‚ 949.5390625# disk.dm-0.out‚23855.94677734375 disk.sda1.out‚0 disk.sda3.out‚ 162.74609375" disk.nvme0c0n1.in‚ 409.17578125# memory_percent‚94.2044734954834 disk.sda3.in‚0 disk.sda.out‚ 162.74609375 disk.sda1.in‚0 disk.dm-1.out‚ 162.74609375( disk.nvme0c0n1.out‚23855.94677734375 disk.drbd1777.out‚0 disk.rbd2.out‚ 126.740234375 disk.dm-0.in‚ 409.17578125 disk.sda.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 409.17578125* proc.memory.percent‚17.531394958496094 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚73.87500714773036% disk./.usageGB‚1100.7909927368164 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.146484375 proc.memory.rssMB‚ 2872.34375 network.recv‚ 1.533647e+06 disk.drbd1777.in‚0 cpu‚25.335990703947385 network.sent‚ 1.219243e+07 disk.sda2.out‚0 disk.rbd2.in‚0²*¦(’Û‚(:‡ Ç åÔÓ²Ù gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.375' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ‚°ü,Ü‚(:¡ Ç åÔ§ë¶ disk.nvme0n1.in‚ 409.17578125% disk.nvme0n1.out‚23955.2333984375! disk.rbd2.out‚126.82666015625 disk.drbd1777.out‚0$ memory_percent‚94.20533180236816* proc.memory.percent‚17.531394958496094" disk.nvme0c0n1.in‚ 409.17578125 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 2872.34375 cpu‚25.334897310236332 network.sent‚ 1.2301569e+07 disk.rbd1.in‚0 disk.dm-0.in‚ 409.17578125" disk.dm-0.out‚23955.2333984375 network.recv‚ 1.540468e+06 disk.dm-1.in‚0' disk.nvme0c0n1.out‚23955.2333984375 disk.sda3.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.87500586769269 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0# disk./.usageGB‚1100.79097366333 disk.dm-1.out‚ 163.52734375 disk.rbd1.out‚ 2.146484375 disk.sda.in‚0 disk.sda.out‚ 163.52734375 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0' proc.memory.availableMB‚ 949.3984375 disk.sda2.out‚0 disk.sda3.out‚ 163.52734375p¢žÝ‚(:’ Ö åÔ—ò gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚88.237' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.647399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°^œ¥.Þ‚(:£ Ö åÔš›§ disk.sda2.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚25.333876721328714% disk./.usageGB‚1100.7911529541016 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda.out‚ 164.1328125 disk.dm-1.out‚ 164.1328125 proc.cpu.threads‚17 network.sent‚ 1.230705e+07( disk.nvme0c0n1.out‚24054.37744140625 disk.rbd3.out‚0( proc.memory.availableMB‚ 949.63671875* proc.memory.percent‚17.531394958496094 disk.rbd2.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 409.17578125 disk.drbd1777.in‚0 disk.dm-0.in‚ 409.17578125 disk.sda1.out‚0$ memory_percent‚94.20387744903564 proc.memory.rssMB‚ 2872.34375& disk.nvme0n1.out‚24054.37744140625 disk.rbd1.in‚0! disk.rbd2.out‚126.85009765625) disk./.usagePercent‚73.87501790004679 disk.nvme0n1.in‚ 409.17578125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 164.1328125# disk.dm-0.out‚24054.37744140625 disk.sda.in‚0 network.recv‚ 1.544254e+06 disk.rbd1.out‚ 2.146484375»±#2ß- _wandbruntime‚4157  _runtime‚4157£“csÉàà ' lr-AdamW/pg1‚0.00015026230170333714 ' lr-AdamW/pg2‚0.00015026230170333714 trainer/global_step‚5499 % _timestamp‚1.7884324723973708e+09 _runtime‚4157.102113227 _step‚448ÛǸ”ÎáÈ trainer/global_step‚5499 % _timestamp‚1.7884324723973708e+09 _runtime‚4157.102113227 _step‚448 ' lr-AdamW/pg1‚0.00015026230170333714 ' lr-AdamW/pg2‚0.00015026230170333714Àÿ'à/2â- _wandbruntime‚4157  _runtime‚4157M\$üá'ãÛO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183546734 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103728024 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103707986 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4600923967488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323840043009536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319239793407488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚319239775404544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323840043009536 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353470971193344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚449 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348644397306368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79818710 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4600923967488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353470777337856 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884324723981256e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26264655 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4826403971584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183526694 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚348644373366272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚319239775404544 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46568710 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323840717374976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26265114 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4583333243556 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79818708 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315465762102948 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚4157.103048048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4600267604992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4826573886976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79818710 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46568708 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310882445898752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72833363 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4600267604992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103707986 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323840717374976 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103728024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392ÝÔ{qæ'äàO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884324723981256e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26264655 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4826403971584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183526694 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚348644373366272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚319239775404544 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46568710 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323840717374976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26265114 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4583333243556 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79818708 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315465762102948 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚4157.103048048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4600267604992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4826573886976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79818710 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46568708 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310882445898752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72833363 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4600267604992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103707986 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323840717374976 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103728024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183546734 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103728024 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103707986 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4600923967488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323840043009536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319239793407488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚319239775404544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323840043009536 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353470971193344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚449 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348644397306368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79818710 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4600923967488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353470777337856ÁæŠ52å- _wandbruntime‚4157  _runtime‚4157ÛË6)äæäM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚323897889854976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚353533478880256 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319296128443392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103746504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348706048044032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4601087032832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183559164 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72846037 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4827269322752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315521482668472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4584149678520 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103746504 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183579230 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79832726 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348706024103936 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315522154145796 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79832726 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4584804116484 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 $ _timestamp‚1.788432473057013e+09 _step‚450 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183579230 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310937332989952 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183559164 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4601743408640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚06¨¥pï&çéM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26269610 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319296146446336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5499 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103726440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4601087032832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚353533293426688 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79832724 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚319296146446336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72846488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183579230 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310937332989952 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183559164 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4601743408640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚323897889854976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚353533478880256 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319296128443392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103746504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348706048044032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4601087032832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183559164 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72846037 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4827269322752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315521482668472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4584149678520 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103746504 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183579230 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79832726 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348706024103936 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315522154145796 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79832726 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4584804116484 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 $ _timestamp‚1.788432473057013e+09 _step‚450 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326698680757760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚351726910249472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26493979 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚322057445349888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318252475447472 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313627652426752 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4641909773312 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73471087 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185153546 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356596587313152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351726934189568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5549 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80519510 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚326699355123200 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4624168582964 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4641909773312 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104654076 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4869653123584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4624823020720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185173586 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80519508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46977110 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚322057427346944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚326699355123200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4641253410816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80519508 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4869493693952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚322057427346944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 % _timestamp‚1.7884325071829815e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104654076 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46977108 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318251803970356 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TVôàO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26493979 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚322057445349888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_poo+%Rø#l.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318252475447472 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313627652426752 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4641909773312 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73471087 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185153546 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356596587313152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351726934189568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5549 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80519510 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚326699355123200 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4624168582964 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4641909773312 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104654076 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4869653123584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4624823020720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185173586 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80519508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46977110 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚322057427346944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚326699355123200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4641253410816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80519508 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4869493693952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚322057427346944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 % _timestamp‚1.7884325071829815e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104654076 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46977108 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318251803970356 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104634038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚322057445349888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4641253410816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80519510 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26494442 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326698680757760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚351726910249472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872Å(\km2õ- _wandbruntime‚4192  _runtime‚4192”“Úê&öäM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80533524 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46985278 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚326755853224448 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80533524 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104672556 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4642072838656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4642729214464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚318308196013204 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚322113780385792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351788560987136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73484218 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4870347010048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185186016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318307524535880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185186016 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚322113780385792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46985276 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4642072838656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104652492 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185206082 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26498940 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚454 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73483736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351788584927232 trainer/global_step‚5549 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4642729214464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313682556557312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4192.543059458 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚326756527603200 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356659101839360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4870516912128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚322113798388736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313682539517952 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185206082 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80533526ü1¢Ýï&÷éM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4642072838656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104652492 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185206082 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26498940 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚454 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73483736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351788584927232 trainer/global_step‚5549 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4642729214464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313682556557312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4192.543059458 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚326756527603200 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356659101839360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4870516912128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚322113798388736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313682539517952 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185206082 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80533526 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80533524 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46985278 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚326755853224448 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80533524 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104672556 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4642072838656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4642729214464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚318308196013204 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚322113780385792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351788560987136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73484218 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4870347010048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185186016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318307524535880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185186016 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚322113780385792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46985276 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080ƪ¨'2ø- _wandbruntime‚4192  _runtime‚4192â+ï\ùÖ 0loss/FRt3/assignment_loss‚3.6915563295284906 2loss/SRqqt3/assignment_loss‚1.4660153855448184 .loss/FBt2/detection_loss‚2.558195360004902 /loss/FBt3/detection_loss‚3.5955931544303894 /loss/FBt4/detection_loss‚2.4656590819358826 1loss/FBt3/assignment_loss‚0.17786723306380736 1loss/FBt4/assignment_loss‚0.07854268630035222 0loss/SRqqt2/detection_loss‚1.614853360905097 _step‚455 0loss/SRqqt3/detection_loss‚2.872138003940168 .loss/FRt2/detection_loss‚2.445798703810064 0loss/FRt2/assignment_loss‚2.8649040649576887 2loss/SRqqt2/assignment_loss‚1.5601065535671437 2loss/SRqqt4/assignment_loss‚1.7171510771594265 1loss/FBt1/assignment_loss‚0.16373628380416147 0loss/FRt1/assignment_loss‚1.0000110256671906 /loss/FRt3/detection_loss‚3.2166709502538047 trainer/global_step‚5549 2loss/SRqqt1/assignment_loss‚0.9479398609201113 1loss/FBt2/assignment_loss‚0.16028932493009052 .loss/FRt1/detection_loss‚1.585169010013342 %loss/total_loss‚43.53782448312958 epoch‚4 1loss/SRqqt4/detection_loss‚1.9438071567903865 /loss/FBt1/detection_loss‚0.5647527075121829 0loss/FRt4/assignment_loss‚3.6152739869223702 .loss/FRt4/detection_loss‚2.219444500075446 1loss/SRqqt1/detection_loss‚1.0123486810922622 $ _timestamp‚1.788432507839473e+09 _runtime‚4192.544104758ò«P,aúÛ 0loss/SRqqt3/detection_loss‚2.872138003940168 .loss/FRt2/detection_loss‚2.445798703810064 0loss/FRt2/assignment_loss‚2.8649040649576887 2loss/SRqqt2/assignment_loss‚1.5601065535671437 2loss/SRqqt4/assignment_loss‚1.7171510771594265 1loss/FBt1/assignment_loss‚0.16373628380416147 0loss/FRt1/assignment_loss‚1.0000110256671906 /loss/FRt3/detection_loss‚3.2166709502538047 trainer/global_step‚5549 2loss/SRqqt1/assignment_loss‚0.9479398609201113 1loss/FBt2/assignment_loss‚0.16028932493009052 .loss/FRt1/detection_loss‚1.585169010013342 %loss/total_loss‚43.53782448312958 epoch‚4 1loss/SRqqt4/detection_loss‚1.9438071567903865 /loss/FBt1/detection_loss‚0.5647527075121829 0loss/FRt4/assignment_loss‚3.6152739869223702 .loss/FRt4/detection_loss‚2.219444500075446 1loss/SRqqt1/detection_loss‚1.0123486810922622 $ _timestamp‚1.788432507839473e+09 _runtime‚4192.544104758 0loss/FRt3/assignment_loss‚3.6915563295284906 2loss/SRqqt3/assignment_loss‚1.4660153855448184 .loss/FBt2/detection_loss‚2.558195360004902 /loss/FBt3/detection_loss‚3.5955931544303894 /loss/FBt4/detection_loss‚2.4656590819358826 1loss/FBt3/assignment_loss‚0.17786723306380736 1loss/FBt4/assignment_loss‚0.07854268630035222 0loss/SRqqt2/detection_loss‚1.614853360905097 _step‚455Ç™ÿ žû‚(:“ ƒ¡åÔŽÇæ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.380599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e.\ ™ü‚(:ž ƒ¡åÔ•—¤& disk.nvme0c0n1.out‚24253.490234375 disk.nvme0n1.in‚ 409.17578125 disk.sda.in‚0) disk./.usagePercent‚73.87500049153446" disk.nvme0c0n1.in‚ 409.17578125 disk.sda2.in‚0 disk.rbd0.in‚0! disk.rbd2.out‚128.58056640625 disk.rbd3.out‚0 network.sent‚ 1.2427579e+07 disk.rbd1.out‚ 2.146484375 disk.rbd2.in‚0 disk.sda3.out‚ 165.76953125•ŠÂb disk.rbd3.in‚0$ disk.nvme0n1.out‚24253.490234375 proc.memory.rssMB‚ 2872.34375 disk.rbd1.in‚0% disk./.usageGB‚1100.7908935546875 disk.sda2.out‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 949.4140625 cpu‚25.3307950363423 disk.sda.out‚ 165.76953125 disk.sda1.out‚0 disk.dm-1.out‚ 165.76953125 network.recv‚ 1.559097e+06 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 409.17578125 disk.sda1.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.531394958496094! disk.dm-0.out‚24253.490234375$ memory_percent‚94.20523643493652 proc.cpu.threads‚17Š‡žî“ý‚(:ˆ ’¡åÔÓ°Ü gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚100.434' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.0868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;½ÿ +þ‚(:  ’¡åÔ±Šç disk.rbd2.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 2.146484375 disk.rbd0.in‚0 disk.sda3.out‚ 166.2421875* proc.memory.percent‚17.531394958496094 disk.nvme0n1.in‚ 409.17578125& disk.nvme0n1.out‚24352.63916015625 disk.rbd1.in‚0 disk.sda2.in‚0! disk.rbd2.out‚128.58056640625 disk.sda.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 409.17578125 network.sent‚ 1.253547e+07 disk.dm-1.out‚ 166.2421875( disk.nvme0c0n1.out‚24352.63916015625 disk.drbd1777.in‚0 cpu‚25.329807848874196 disk.sda.out‚ 166.2421875 disk.sda1.in‚0 disk.drbd1777.out‚0# disk.dm-0.out‚24352.63916015625 disk.sda3.in‚0# memory_percent‚94.2082405090332 network.recv‚ 1.564986e+06% disk./.usageGB‚1100.7911376953125" disk.nvme0c0n1.in‚ 409.17578125 disk.sda2.out‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 948.921875 proc.memory.rssMB‚ 2872.34375) disk./.usagePercent‚73.87501687601666 disk.dm-1.in‚0 disk.rbd0.out‚0VëÆ2ÿ- _wandbruntime‚4226  _runtime‚4226lÚ6²É€Ã ' lr-AdamW/pg2‚0.00014715461204731347 trainer/global_step‚5599 % _timestamp‚1.7884325422278972e+09 _runtime‚4226.932738598 _step‚456 ' lr-AdamW/pg1‚0.00014715461204731347Rc>ÀÎÈ _step‚456 ' lr-AdamW/pg1‚0.00014715461204731347 ' lr-AdamW/pg2‚0.00014715461204731347 trainer/global_step‚5599 % _timestamp‚1.7884325422278972e+09 _runtime‚4226.932738598ÈùËø2‚- _wandbruntime‚4226  _runtime‚4226óïCá'ƒÛO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321037845837764 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚316372841915392 % _timestamp‚1.7884325422286613e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81220308 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324875097292288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74109330 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4665003922372 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚5599 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105560090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26723369 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4665658360128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81220310 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329557318505984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚457 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329557318505984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354809447132672 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74108877 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4682895579136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105560090 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329557992871424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186800438 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4912836025344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4682895579136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47385508 _runtime‚4226.933584251 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚324875097292288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4912672401408!ÿýæ'„àO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4912672401408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321037845837764 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚316372841915392 % _timestamp‚1.7884325422286613e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81220308 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324875097292288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74109330 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4665003922372 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚5599 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105560090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26723369 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4665658360128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81220310 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329557318505984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚457 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329557318505984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354809447132672 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74108877 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4682895579136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105560090 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329557992871424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186800438 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4912836025344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4682895579136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47385508 _runtime‚4226.933584251 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚324875097292288ÉÒÌÁð2…- _wandbruntime‚4227  _runtime‚4227ÜŸ Óq%†åM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324931432328192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74121542 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186832934 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4913703804416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884325428842635e+09 _runtime‚4227.589614826 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚458 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186812868 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105578544 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105598608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329615165351424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186832934 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26728317 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329614490972672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105578544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26727866 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4665820357336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186812868 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81234324 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324931450331136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324931432328192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354871097870336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105598608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81234324 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4683715020288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359784825614848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329615165351424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47393676 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0  }ð&‡êM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105598608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329615165351424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186832934 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26728317 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329614490972672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105578544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26727866 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4665820357336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186812868 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81234324 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324931450331136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324931432328192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354871097870336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105598608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81234324 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4683715020288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359784825614848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329615165351424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47393676 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324931432328192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74121542 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186832934 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4913703804416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884325428842635e+09 _runtime‚4227.589614826 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚458 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186812868 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105578544 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0ÊP8Ž2ˆ- _wandbruntime‚4227  _runtime‚4227¥Ë8\‰Ö % _timestamp‚1.7884325428859591e+09 0loss/FRt3/assignment_loss‚3.5069812509146603 2loss/SRqqt1/assignment_loss‚1.0828550680397222 2loss/SRqqt3/assignment_loss‚1.9728401561780853 1loss/FBt1/assignment_loss‚0.15471458667832186 /loss/FRt3/detection_loss‚2.3195317658511074 0loss/FRt4/assignment_loss‚3.5232741411139323 /loss/FRt1/detection_loss‚1.9407932331616229 0loss/SRqqt4/detection_loss‚1.922101636540215 .loss/FBt4/detection_loss‚2.309426635503769 1loss/SRqqt2/assignment_loss‚1.455013173125815 /loss/FRt2/detection_loss‚2.4187847311082096 /loss/FBt3/detection_loss‚2.9154162406921387 _runtime‚4227.590607126 0loss/SRqqt2/detection_loss‚1.919091708827437 epoch‚4 trainer/global_step‚5599 0loss/FRt1/assignment_loss‚1.5336433692411944 1loss/FBt4/assignment_loss‚0.48081196495331824 /loss/FBt1/detection_loss‚0.7857983774827978 1loss/FBt2/assignment_loss‚0.33521869491475326 1loss/SRqqt3/detection_loss‚2.1169174197985203 %loss/total_loss‚44.38024022861104 _step‚459 0loss/FRt2/assignment_loss‚3.1685684789781985 2loss/SRqqt4/assignment_loss‚1.7851610708770886 1loss/FBt3/assignment_loss‚0.15656174602918327 .loss/FRt4/detection_loss‚2.319710232862612 1loss/SRqqt1/detection_loss‚1.2731948686333803 -loss/FBt2/detection_loss‚2.98382967710495^ÂXaŠÛ 0loss/FRt1/assignment_loss‚1.5336433692411944 1loss/FBt4/assignment_loss‚0.48081196495331824 /loss/FBt1/detection_loss‚0.7857983774827978 1loss/FBt2/assignment_loss‚0.33521869491475326 1loss/SRqqt3/detection_loss‚2.1169174197985203 %loss/total_loss‚44.38024022861104 _step‚459 0loss/FRt2/assignment_loss‚3.1685684789781985 2loss/SRqqt4/assignment_loss‚1.7851610708770886 1loss/FBt3/assignment_loss‚0.15656174602918327 .loss/FRt4/detection_loss‚2.319710232862612 1loss/SRqqt1/detection_loss‚1.2731948686333803 -loss/FBt2/detection_loss‚2.98382967710495 % _timestamp‚1.7884325428859591e+09 0loss/FRt3/assignment_loss‚3.5069812509146603 2loss/SRqqt1/assignment_loss‚1.0828550680397222 2loss/SRqqt3/assignment_loss‚1.9728401561780853 1loss/FBt1/assignment_loss‚0.15471458667832186 /loss/FRt3/detection_loss‚2.3195317658511074 0loss/FRt4/assignment_loss‚3.5232741411139323 /loss/FRt1/detection_loss‚1.9407932331616229 0loss/SRqqt4/detection_loss‚1.922101636540215 .loss/FBt4/detection_loss‚2.309426635503769 1loss/SRqqt2/assignment_loss‚1.455013173125815 /loss/FRt2/detection_loss‚2.4187847311082096 /loss/FBt3/detection_loss‚2.9154162406921387 _runtime‚4227.590607126 0loss/SRqqt2/detection_loss‚1.919091708827437 epoch‚4 trainer/global_step‚5599ËË7€Ðž‹‚(:“ ¡¡åÔšÃð gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.093' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.818600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚„v,Œ‚(:¡ ¡¡åÔÃÆŠ‘ disk.sda3.out‚ 166.83984375 disk.dm-1.out‚ 166.83984375$ memory_percent‚94.21114921569824' proc.memory.availableMB‚ 948.4453125 proc.memory.rssMB‚ 2872.34375 disk.rbd2.out‚128.6826171875 disk.sda.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 2.146484375" disk.nvme0c0n1.in‚ 409.17578125 disk.drbd1777.in‚0 disk.dm-0.in‚ 409.17578125 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 166.83984375 network.recv‚ 1.56971e+06 disk.sda2.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 409.17578125% disk.nvme0n1.out‚24352.9521484375' disk.nvme0c0n1.out‚24352.9521484375 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0) disk./.usagePercent‚73.87504759692077% disk./.usageGB‚1100.7915954589844 disk.sda2.in‚0" disk.dm-0.out‚24352.9521484375* proc.memory.percent‚17.531394958496094 network.sent‚ 1.2541795e+07 disk.rbd3.in‚0 cpu‚25.328805951329915 disk.rbd0.out‚0 disk.rbd0.in‚0ãçؽž‚(:“ °¡åԿ¿ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.216' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.043200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y*wÌ,Ž‚(:¡ °¡åÔúΑ disk.dm-1.out‚ 167.27734375 disk.dm-0.in‚ 409.17578125 disk.sda.in‚0 cpu‚25.327820185561386" disk.nvme0c0n1.in‚ 409.17578125 disk.rbd3.in‚0# disk.dm-0.out‚24452.50146484375 network.recv‚ 1.57483e+06& disk.nvme0n1.out‚24452.50146484375 disk.dm-1.in‚0% proc.memory.availableMB‚ 949.15625 proc.memory.rssMB‚ 2872.34375 disk.rbd1.in‚0( disk.nvme0c0n1.out‚24452.50146484375 disk.rbd2.out‚130.1943359375 disk.sda3.in‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.531394958496094 disk.rbd1.out‚ 2.146484375 disk.rbd0.in‚0 disk.sda3.out‚ 167.27734375 disk.sda1.out‚0# memory_percent‚94.2068099975586% disk./.usageGB‚1100.7916717529297 disk.nvme0n1.in‚ 409.17578125 disk.rbd3.out‚0 disk.sda1.in‚0 network.sent‚ 1.2649063e+07 disk.rbd2.in‚0 disk.sda.out‚ 167.27734375 proc.cpu.threads‚17) disk./.usagePercent‚73.87505271707145 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0…Pˆž‚(:“ ¿¡åÔ‹¼ü gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.075' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.414999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„@)‚(:ž ¿¡åÔ¾ú— disk.dm-1.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 409.17578125 network.sent‚ 1.2655325e+07) disk./.usagePercent‚73.87505143703378 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 cpu‚25.32680698275628 disk.sda.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚24551.57958984375 disk.drbd1777.out‚0 disk.dm-1.out‚ 167.8671875" disk.nvme0c0n1.in‚ 409.17578125% proc.memory.availableMB‚ 949.15625% disk./.usageGB‚1100.7916526794434 disk.rbd1.out‚ 2.146484375 disk.sda2.out‚0 disk.rbd2.out‚130.3349609375 proc.memory.rssMB‚ 2872.34375 disk.sda3.out‚ 167.8671875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 409.17578125& disk.nvme0n1.out‚24551.57958984375 disk.rbd1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.531394958496094 disk.sda1.in‚0( disk.nvme0c0n1.out‚24551.57958984375# memory_percent‚94.2068099975586 disk.rbd0.out‚0 disk.sda.out‚ 167.8671875 network.recv‚ 1.579228e+06 disk.sda3.in‚0 disk.sda2.in‚0_X#2‘- _wandbruntime‚4261  _runtime‚4261$ãŒÉ’à ' lr-AdamW/pg1‚0.00014559002561452376 ' lr-AdamW/pg2‚0.00014559002561452376 trainer/global_step‚5649 % _timestamp‚1.7884325770983763e+09 _runtime‚4261.803099932 _step‚460ó(œÎ“È trainer/global_step‚5649 % _timestamp‚1.7884325770983763e+09 _runtime‚4261.803099932 _step‚460 ' lr-AdamW/pg1‚0.00014559002561452376 ' lr-AdamW/pg2‚0.00014559002561452376ÌãÆóü2”- _wandbruntime‚4261  _runtime‚4261ÇBD§à'•ÚO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚332416630619648 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81921110 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81921110 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188407250 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4723225022464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81921108 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81921108 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106506180 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323824559182288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327692749234688 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362847998595584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74746402 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4706493699536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323823887705172 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚332416630619648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106486142 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4723225022464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4705839261780 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188427290 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327692749234688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327692731231744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26952492 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332415956254208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319118048443392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106506180 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944_‰Þú–ßO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81921110 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81921110 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188407250 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4723225022464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81921108 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81921108 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106506180 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323824559182288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327692749234688 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362847998595584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74746402 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4706493699536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323823887705172 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚332416630619648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106486142 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4723225022464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4705839261780 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188427290 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327692749234688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327692731231744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26952492 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332415956254208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319118048443392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106506180 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚332416630619648 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872Íç­Ê€2—- _wandbruntime‚4262  _runtime‚4262Ý91ë&˜åM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47802076 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4956669752832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362910304506368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362910502542848 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4724044450304 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81935124 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188459786 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106524660 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106524660 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4707310134708 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327749102273536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4706655696744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26956464 % _timestamp‚1.7884325777577655e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327749084270592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188439720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81935124 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106504596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚332473128720896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81935126 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319172952573952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚462 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323880279748020 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4724700826112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81935126 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74758540p÷¦Qð&™êM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚332473803099648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4956843849216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188439720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26956916 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188459786 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327749102273536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4706655696744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26956464 % _timestamp‚1.7884325777577655e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327749084270592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188439720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81935124 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106504596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚332473128720896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81935126 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319172952573952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚462 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323880279748020 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4724700826112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81935126 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74758540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47802076 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4956669752832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362910304506368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362910502542848 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4724044450304 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81935124 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188459786 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106524660 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106524660 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4707310134708 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335274594002432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.onÌŲ¥—%_train_batch_start/active.all.freed‚ 190034102 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190054142 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4746674601188 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚365973731486720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321863254971392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107432232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335275268367872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884326120003667e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27182089 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4999186647552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4764867190784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚465 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321863272010752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚330510401177088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚360974544839168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335274594002432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27182550 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4747329038944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82621910 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚330510383174144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4999020926464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚365973541825536 trainer/global_step‚5699 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335275268367872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75384860 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚4296.705230231 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82621908 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚330510401177088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4764867190784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107412194 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107412194 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680X«|¤æ'¦àO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75384860 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚4296.705230231 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82621908 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚330510401177088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4764867190784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107412194 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107412194 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82621908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48202310 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326609929572580 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335274594002432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190034102 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190054142 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4746674601188 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚365973731486720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321863254971392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107432232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335275268367872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884326120003667e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27182089 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4999186647552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4764867190784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚465 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321863272010752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚330510401177088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚360974544839168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335274594002432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27182550 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4747329038944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82621910 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚330510383174144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4999020926464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚365973541825536 trainer/global_step‚5699 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335275268367872ÑPxb2§- _wandbruntime‚4297  _runtime‚4297wQzEë&¨åM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚330566736212992 % _timestamp‚1.7884326126542482e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 _step‚466 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335332440847872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4765030256128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48210478 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4765686631936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4747491036152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82635926 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48210476 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190086638 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚330566736212992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190086638 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107450712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75397665 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190066572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75397178 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361036171636736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚330566754215936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366036239672320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361036195576832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107430648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366036041635840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4765030256128 trainer/global_step‚5699 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82635924 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27187187 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5000044095488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335332440847872©Q)C ©êM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48210476 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190086638 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚330566736212992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190086638 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107450712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75397665 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190066572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_Ú­allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75397178 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361036171636736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚330566754215936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366036239672320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361036195576832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107430648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366036041635840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4765030256128 trainer/global_step‚5699 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82635924 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27187187 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5000044095488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335332440847872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚330566736212992 % _timestamp‚1.7884326126542482e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 _step‚466 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335332440847872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4765030256128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48210478 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4765686631936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4747491036152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82635926 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952ÒÒŒZ2ª- _wandbruntime‚4297  _runtime‚4297»cR¶[«Õ /loss/FBt4/detection_loss‚3.1002781987190247 % _timestamp‚1.7884326126557715e+09 _step‚467 0loss/FRt1/assignment_loss‚1.3912228732778316 1loss/SRqqt2/assignment_loss‚1.871716506332159 2loss/SRqqt3/assignment_loss‚1.8809896252221532 2loss/SRqqt4/assignment_loss‚1.2559675128530647 /loss/FRt2/detection_loss‚2.4471041726511578 trainer/global_step‚5699 _runtime‚4297.360398687 /loss/FRt4/assignment_loss‚3.397893228429429 1loss/FBt1/assignment_loss‚0.08237263204103264 %loss/total_loss‚43.99655003980569 /loss/FBt1/detection_loss‚0.7922000764192988 0loss/FRt2/assignment_loss‚3.4413286613863567 1loss/FBt3/assignment_loss‚0.36988086050841956 -loss/FRt1/detection_loss‚1.76285602307568 /loss/FRt4/detection_loss‚2.3247496054527605 /loss/SRqqt3/detection_loss‚2.12021576695972 0loss/FRt3/assignment_loss‚3.0002815204281963 1loss/FBt2/assignment_loss‚0.22848685298716495 1loss/SRqqt1/detection_loss‚0.9616641965694726 /loss/FRt3/detection_loss‚2.6782173379775016 /loss/FBt3/detection_loss‚2.5073936303456623 epoch‚4 2loss/SRqqt1/assignment_loss‚1.2486707791758818 1loss/FBt4/assignment_loss‚0.18853970670274325 0loss/SRqqt2/detection_loss‚2.003603113591671 1loss/SRqqt4/detection_loss‚1.6481493458909504 .loss/FBt2/detection_loss‚3.292767812808355’VÓ`¬Ú 0loss/FRt2/assignment_loss‚3.4413286613863567 1loss/FBt3/assignment_loss‚0.36988086050841956 -loss/FRt1/detection_loss‚1.76285602307568 /loss/FRt4/detection_loss‚2.3247496054527605 /loss/SRqqt3/detection_loss‚2.12021576695972 0loss/FRt3/assignment_loss‚3.0002815204281963 1loss/FBt2/assignment_loss‚0.22848685298716495 1loss/SRqqt1/detection_loss‚0.9616641965694726 /loss/FRt3/detection_loss‚2.6782173379775016 /loss/FBt3/detection_loss‚2.5073936303456623 epoch‚4 2loss/SRqqt1/assignment_loss‚1.2486707791758818 1loss/FBt4/assignment_loss‚0.18853970670274325 0loss/SRqqt2/detection_loss‚2.003603113591671 1loss/SRqqt4/detection_loss‚1.6481493458909504 .loss/FBt2/detection_loss‚3.292767812808355 /loss/FBt4/detection_loss‚3.1002781987190247 % _timestamp‚1.7884326126557715e+09 _step‚467 0loss/FRt1/assignment_loss‚1.3912228732778316 1loss/SRqqt2/assignment_loss‚1.871716506332159 2loss/SRqqt3/assignment_loss‚1.8809896252221532 2loss/SRqqt4/assignment_loss‚1.2559675128530647 /loss/FRt2/detection_loss‚2.4471041726511578 trainer/global_step‚5699 _runtime‚4297.360398687 /loss/FRt4/assignment_loss‚3.397893228429429 1loss/FBt1/assignment_loss‚0.08237263204103264 %loss/total_loss‚43.99655003980569 /loss/FBt1/detection_loss‚0.7922000764192988Ó‚wS}ž­‚(:“ ì¡åÔ€âÐŽ gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.201' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.440199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ð=Ä+®‚(:  ì¡åÔÎô»‘ proc.memory.rssMB‚ 2872.34375 cpu‚25.323857627137304 disk.dm-1.in‚0' disk.nvme0c0n1.out‚24751.2216796875 disk.rbd0.out‚0 disk.dm-1.out‚ 169.3046875 disk.nvme0n1.in‚ 409.17578125% disk.nvme0n1.out‚24751.2216796875 disk.rbd1.out‚ 2.146484375" disk.nvme0c0n1.in‚ 409.17578125 disk.sda3.in‚0 disk.sda3.out‚ 169.3046875 disk.drbd1777.out‚0 network.recv‚ 1.593879e+06 disk.rbd3.out‚0 disk.dm-0.in‚ 409.17578125 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.531394958496094( disk./.usagePercent‚73.8750081717605 disk.rbd0.in‚0 disk.drbd1777.in‚0( proc.memory.availableMB‚ 948.76171875 proc.cpu.threads‚17% disk./.usageGB‚1100.7910079956055 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda.out‚ 169.3046875! disk.rbd2.out‚130.43701171875 network.sent‚ 1.2774928e+07" disk.dm-0.out‚24751.2216796875 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚94.20921802520752†AÜ“¯‚(:ˆ û¡åÔðöñ gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.779' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.3558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?Èd-°‚(:¢ û¡åÔÐΦ‘' disk.nvme0c0n1.out‚24850.3603515625% disk./.usageGB‚1100.7909317016602 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0* proc.memory.percent‚17.531394958496094 network.recv‚ 1.600119e+06 disk.sda3.out‚ 170.00390625" disk.dm-0.out‚24850.3603515625 disk.sda.out‚ 170.00390625% disk.nvme0n1.out‚24850.3603515625$ memory_percent‚94.20857429504395 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚0 proc.memory.rssMB‚ 2872.34375) disk./.usagePercent‚73.87500305160981 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 948.8671875 disk.nvme0n1.in‚ 409.17578125 disk.rbd0.out‚0! disk.rbd2.out‚130.51513671875 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 409.17578125 disk.dm-1.out‚ 170.00390625 disk.rbd1.in‚0 cpu‚25.32291450214931 proc.cpu.threads‚17 network.sent‚ 1.2883723e+07 disk.rbd1.out‚ 2.146484375" disk.nvme0c0n1.in‚ 409.17578125Sœè/2±- _wandbruntime‚4331  _runtime‚4331åɲà ' lr-AdamW/pg1‚0.00014244099975877636 ' lr-AdamW/pg2‚0.00014244099975877636 trainer/global_step‚5749 % _timestamp‚1.7884326468944252e+09 _runtime‚4331.599143982 _step‚468ÔÔwÎ³È ' lr-AdamW/pg1‚0.00014244099975877636 ' lr-AdamW/pg2‚0.00014244099975877636 trainer/global_step‚5749 % _timestamp‚1.7884326468944252e+09 _runtime‚4331.599143982 _step‚468ÔÏCð2´- _wandbruntime‚4331  _runtime‚4331ôá'µÛO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333328053119488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108358284 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27412650 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4788164378352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191660954 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27412177 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83322710 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364057057782272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚469 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4805196634112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369099355472896 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4805852996608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329395971439988 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83322708 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333328053119488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191680994 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48610710 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338133231750656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884326468951545e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108338246 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324608461499392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4787509940596 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 trainer/global_step‚5749 _runtime‚4331.600107055 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329396642917104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108358284 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚338133231750656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76022885 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333328035116544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4805196634112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83322708 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚475 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206438400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369099149034496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182498304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83322710 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206438400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369099149034496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182498304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83322710 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83322710 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364057057782272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚469 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4805196634112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369099355472896 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4805852996608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329395971439988 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83322708 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]IDeviceStatsMonitÜkœ€Gor.on_train_batch_start/active_bytes.large_pool.allocated‚333328053119488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191680994 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48610710 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338133231750656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884326468951545e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108338246ÕËnzŒ2·- _wandbruntime‚4332  _runtime‚4332´ÑLê&¸äM trainer/global_step‚5749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83336724 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4806016061952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191693424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369161846144000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338190404217344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333384406158336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚470 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108376764 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191713490 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4806672437760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108356700 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5043113683968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4806672437760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191693424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884326477721298e+09 _runtime‚ 4332.47746998 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333384406158336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329452363482836 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4806016061952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83336724 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76035570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48618876 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76036021 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27417143 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48618878 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369161650204672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364118708519936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4788980813524 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83336726 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83336726 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333384388155392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108356700 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338191078596096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844áÍâï&¹éM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83336726 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83336726 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333384388155392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108356700 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338191078596096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚5749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83336724 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4806016061952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191693424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369161846144000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338190404217344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333384406158336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚470 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108376764 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191713490 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4806672437760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108356700 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5043113683968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4806672437760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191693424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884326477721298e+09 _runtime‚ 4332.47746998 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333384406158336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329452363482836 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4806016061952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83336724 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76035570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48618876 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76036021 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27417143 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48618878 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369161650204672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364118708519936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4788980813524ÖIš6ò2º- _wandbruntime‚4332  _runtime‚4332™è–áX»Ò 0loss/SRqqt1/detection_loss‚1.037208620745402 /loss/FBt2/detection_loss‚2.5016710228390164 %loss/total_loss‚42.79081773441419 0loss/FRt4/assignment_loss‚3.1604903427263102 2loss/SRqqt2/assignment_loss‚1.4826472838493912 1loss/FBt1/assignment_loss‚0.24692744229379343 1loss/FBt2/assignment_loss‚0.21387570464543793 /loss/FRt1/detection_loss‚1.8006780025150095 % _timestamp‚1.7884326477741244e+09 /loss/FBt1/detection_loss‚0.3865769340410932 .loss/FBt4/detection_loss‚2.762765089670817 epoch‚4 trainer/global_step‚5749 _step‚471 /loss/FRt1/assignment_loss‚1.514255939494996 2loss/SRqqt4/assignment_loss‚1.3988897187964004 -loss/FRt2/detection_loss‚2.70325291962237 1loss/SRqqt4/detection_loss‚1.8276800957380557 /loss/FBt3/detection_loss‚2.2928594992710996 0loss/FRt3/assignment_loss‚2.9841132889623228 1loss/SRqqt2/detection_loss‚1.7316800632260063 2loss/SRqqt1/assignment_loss‚0.9692790214832012 0loss/FBt3/assignment_loss‚0.5341762241083555 1loss/SRqqt3/detection_loss‚2.3771839792078193 _runtime‚4332.478780347 /loss/FRt2/assignment_loss‚2.919046323041658 2loss/SRqqt3/assignment_loss‚1.9558746787634762 0loss/FBt4/assignment_loss‚0.3305449715505044 -loss/FRt4/detection_loss‚2.35376582108438 .loss/FRt3/detection_loss‚3.305374746737273rŸB]¼× 0loss/FRt4/assignment_loss‚3.1604903427263102 2loss/SRqqt2/assignment_loss‚1.4826472838493912 1loss/FBt1/assignment_loss‚0.24692744229379343 1loss/FBt2/assignment_loss‚0.21387570464543793 /loss/FRt1/detection_loss‚1.8006780025150095 % _timestamp‚1.7884326477741244e+09 /loss/FBt1/detection_loss‚0.3865769340410932 .loss/FBt4/detection_loss‚2.762765089670817 epoch‚4 trainer/global_step‚5749 _step‚471 /loss/FRt1/assignment_loss‚1.514255939494996 2loss/SRqqt4/assignment_loss‚1.3988897187964004 -loss/FRt2/detection_loss‚2.70325291962237 1loss/SRqqt4/detection_loss‚1.8276800957380557 /loss/FBt3/detection_loss‚2.2928594992710996 0loss/FRt3/assignment_loss‚2.9841132889623228 1loss/SRqqt2/detection_loss‚1.7316800632260063 2loss/SRqqt1/assignment_loss‚0.9692790214832012 0loss/FBt3/assignment_loss‚0.5341762241083555 1loss/SRqqt3/detection_loss‚2.3771839792078193 _runtime‚4332.478780347 /loss/FRt2/assignment_loss‚2.919046323041658 2loss/SRqqt3/assignment_loss‚1.9558746787634762 0loss/FBt4/assignment_loss‚0.3305449715505044 -loss/FRt4/detection_loss‚2.35376582108438 .loss/FRt3/detection_loss‚3.305374746737273 0loss/SRqqt1/detection_loss‚1.037208620745402 /loss/FBt2/detection_loss‚2.5016710228390164 %loss/total_loss‚42.79081773441419וTnÒ“½‚(:ˆ Š¢åÔþšÊ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚150.672' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.1344 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ïú1¾‚(:¦ Š¢åÔ·©û! disk.rbd2.out‚132.02685546875 disk.drbd1777.in‚0 proc.memory.rssMB‚ 2872.34375# disk.dm-0.out‚24950.04052734375 disk.rbd2.in‚0 cpu‚25.321991905451107 disk.dm-0.in‚ 409.17578125 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 170.67578125 disk.sda2.in‚0 disk.sda3.out‚ 170.67578125 disk.drbd1777.out‚0) disk./.usagePercent‚73.87505169304131 disk.sda.in‚0 disk.sda.out‚ 170.67578125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 409.17578125 disk.rbd3.in‚0$ memory_percent‚94.20638084411621* proc.memory.percent‚17.531394958496094 network.sent‚ 1.2889067e+07 disk.sda1.in‚0' proc.memory.availableMB‚ 949.2265625 proc.cpu.threads‚17& disk.nvme0n1.out‚24950.04052734375 disk.rbd0.out‚0 disk.rbd3.out‚0 network.recv‚ 1.604091e+06% disk./.usageGB‚1100.7916564941406( disk.nvme0c0n1.out‚24950.04052734375 disk.sda3.in‚0 disk.rbd1.out‚ 2.146484375" disk.nvme0c0n1.in‚ 409.17578125 disk.sda2.out‚0 disk.rbd0.in‚0/§Vž¿‚(:“ ™¢åÔáÏô gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚155.216' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.043200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û”0À‚(:¥ ™¢åÔ‹™è‘ network.sent‚ 1.299812e+07 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0! disk.rbd2.out‚132.10498046875# disk.dm-0.out‚25049.14208984375 proc.memory.rssMB‚ 2872.34375 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 948.73046875" disk.nvme0c0n1.in‚ 409.17578125) disk./.usagePercent‚73.87505041300363 disk.sda1.in‚0 disk.dm-1.out‚ 171.32421875 disk.rbd1.out‚ 2.232421875 disk.rbd0.out‚0# memory_percent‚94.2094087600708 cpu‚25.320980207576856% disk./.usageGB‚1100.7916374206543 disk.sda.in‚0( disk.nvme0c0n1.out‚25049.14208984375 network.recv‚ 1.610548e+06 disk.sda3.out‚ 171.32421875& disk.nvme0n1.out‚25049.14208984375 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 409.17578125 disk.sda.out‚ 171.32421875 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 409.17578125* proc.memory.percent‚17.531394958496094 proc.cpu.threads‚17¡¼EqžÁ‚(:“ ¨¢åÔØÍÚ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.967' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.593400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å¿}I2‚(:§ ¨¢åÔÈÖä disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.3003591e+07 disk.drbd1777.out‚0 disk.dm-0.in‚ 409.17578125 disk.rbd1.out‚ 2.232421875 disk.rbd3.in‚0 disk.sda.out‚ 171.75390625 disk.dm-1.in‚0 proc.memory.rssMB‚ 2872.34375 disk.drbd1777.in‚0# disk.dm-0.out‚25049.56396484375& disk.nvme0n1.out‚25049.56396484375( disk.nvme0c0n1.out‚25049.56396484375 disk.rbd0.in‚0$ memory_percent‚94.20788288116455% disk./.usageGB‚1100.7916946411133 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 409.17578125! disk.rbd2.out‚132.10498046875 cpu‚25.320078793860677 disk.sda3.out‚ 171.75390625 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0) disk./.usagePercent‚73.87505425311666 disk.dm-1.out‚ 171.75390625( proc.memory.availableMB‚ 948.98046875* proc.memory.percent‚17.531394958496094 network.recv‚ 1.614711e+06 disk.nvme0n1.in‚ 409.17578125•€2Ã- _wandbruntime‚4366  _runtime‚43664à ÈÄ ' lr-AdamW/pg1‚0.00014085721658808716 ' lr-AdamW/pg2‚0.00014085721658808716 trainer/global_step‚5799 % _timestamp‚1.7884326818695488e+09 _runtime‚ 4366.57434052 _step‚472­·`¶ÍÅÇ % _timestamp‚1.7884326818695488e+09 _runtime‚ 4366.57434052 _step‚472 ' lr-AdamW/pg1‚0.00014085721658808716 ' lr-AdamW/pg2‚0.00014085721658808716 trainer/global_step‚5799Ø +Ç2Æ- _wandbruntime‚4366  _runtime‚4366;•GáîÇØO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372225050231296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27641422 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚340992543864320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84023508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109284336 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4846182439936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚332182013307396 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193287806 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5085278487552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336145705061888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84023510 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5085431625728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336145705061888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27641875 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340991869498880 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84023508 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _step‚473‘,úåã'ÈÝO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27641875 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340991869498880 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84023508 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _step‚473 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372225050231296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27641422 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚340992543864320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84023508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109284336 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4846182439936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚332182013307396 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193287806 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5085278487552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336145705061888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84023510 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5085431625728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336145705061888Ù<£Æn2É- _wandbruntime‚4367  _runtime‚4367鉂ë&ÊåM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336202040097792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76673739 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76673278 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372287568341504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332238405350244 trainer/global_step‚5799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193320276 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84037524 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4847001867776 _step‚474 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4847658243584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193340342 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109282752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49027278 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27646002 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4829816152932 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84037526 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336202058100736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚327408589197312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341049041965568 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84037526 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4829161714968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193340342 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372287372402176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4847001867776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336202058100736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884326825251806e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327408572157952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341049716344320 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341049716344320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193320276 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367201269343232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5086298998272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚332237733872920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84037524 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27646461 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944@öŽÔíËêM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341049041965568 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84037526 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4829161714968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193340342 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372287372402176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4847001867776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336202058100736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884326825251806e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327408572157952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341049716344320 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341049716344320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193320276 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367201269343232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5086298998272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚332237733872920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84037524 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27646461 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 33²}ç 6208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336202040097792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76673739 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76673278 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372287568341504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332238405350244 trainer/global_step‚5799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193320276 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84037524 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4847001867776 _step‚474 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4847658243584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193340342 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109282752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49027278 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27646002 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4829816152932 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84037526 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336202058100736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚327408589197312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0Ú :m±2Ì- _wandbruntime‚4367  _runtime‚4367±^^ÍØ /loss/FRt4/assignment_loss‚4.165111983144605 $ _timestamp‚1.788432682526794e+09 _runtime‚4367.231421103 0loss/FBt2/assignment_loss‚0.1335529390967817 /loss/FRt1/detection_loss‚1.7158956789196684 1loss/SRqqt3/detection_loss‚2.4727085886093287 1loss/SRqqt4/detection_loss‚1.7471791163403938 /loss/FBt2/detection_loss‚2.5842077817235674 &loss/total_loss‚45.711328673509314 2loss/SRqqt2/assignment_loss‚1.7219490440717589 2loss/SRqqt4/assignment_loss‚1.8348868966419647 1loss/FBt4/assignment_loss‚0.22841572442225047 /loss/FRt3/detection_loss‚2.7908669897488187 /loss/FBt4/detection_loss‚2.7586581025804793 epoch‚4 2loss/SRqqt1/assignment_loss‚0.8631052006967366 .loss/FRt4/detection_loss‚2.318544617375812 1loss/SRqqt1/detection_loss‚0.7811496560979221 trainer/global_step‚5799 0loss/FRt1/assignment_loss‚1.3657916268477073 /loss/FRt3/assignment_loss‚3.930780989783151 1loss/SRqqt3/assignment_loss‚1.942877089174894 1loss/FBt3/assignment_loss‚0.23996126351412386 /loss/FBt3/detection_loss‚3.3616693139076235 /loss/FRt2/detection_loss‚2.4459600883722303 _step‚475 0loss/FRt2/assignment_loss‚3.5057899284362795 1loss/FBt1/assignment_loss‚0.18803331621132613 1loss/SRqqt2/detection_loss‚1.9552485005309184 /loss/FBt1/detection_loss‚0.6589842372609732±ùcÎÝ 0loss/FRt1/assignment_loss‚1.3657916268477073 /loss/FRt3/assignment_loss‚3.930780989783151 1loss/SRqqt3/assignment_loss‚1.942877089174894 1loss/FBt3/assignment_loss‚0.23996126351412386 /loss/FBt3/detection_loss‚3.3616693139076235 /loss/FRt2/detection_loss‚2.4459600883722303 _step‚475 0loss/FRt2/assignment_loss‚3.5057899284362795 1loss/FBt1/assignment_loss‚0.18803331621132613 1loss/SRqqt2/detection_loss‚1.9552485005309184 /loss/FBt1/detection_loss‚0.6589842372609732 /loss/FRt4/assignment_loss‚4.165111983144605 $ _timestamp‚1.788432682526794e+09 _runtime‚4367.231421103 0loss/FBt2/assignment_loss‚0.1335529390967817 /loss/FRt1/detection_loss‚1.7158956789196684 1loss/SRqqt3/detection_loss‚2.4727085886093287 1loss/SRqqt4/detection_loss‚1.7471791163403938 /loss/FBt2/detection_loss‚2.5842077817235674 &loss/total_loss‚45.711328673509314 2loss/SRqqt2/assignment_loss‚1.7219490440717589 2loss/SRqqt4/assignment_loss‚1.8348868966419647 1loss/FBt4/assignment_loss‚0.22841572442225047 /loss/FRt3/detection_loss‚2.7908669897488187 /loss/FBt4/detection_loss‚2.7586581025804793 epoch‚4 2loss/SRqqt1/assignment_loss‚0.8631052006967366 .loss/FRt4/detection_loss‚2.318544617375812 1loss/SRqqt1/detection_loss‚0.7811496560979221 trainer/global_step‚5799ÛÒÿٞς(:“ ·¢åÔ®«Ž gpu.0.gpu‚44.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚143.831' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.766199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž7+c*Ђ(:Ÿ ·¢åÔƒóº network.sent‚ 1.3009859e+07 disk.dm-0.in‚ 409.17578125 disk.sda.out‚ 172.4765625& disk.nvme0n1.out‚25152.14599609375( proc.memory.availableMB‚ 948.66796875 proc.memory.rssMB‚ 2872.34375) disk./.usagePercent‚73.87525957115912 disk.rbd0.in‚0 disk.rbd1.in‚0 network.recv‚ 1.619228e+06 disk.sda2.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1100.7947540283203 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 2.294921875 disk.sda3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚25152.14599609375 disk.rbd2.out‚ 132.203125# disk.dm-0.out‚25152.14599609375 disk.dm-1.in‚0 disk.nvme0n1.in‚ 409.17578125 disk.sda3.out‚ 172.4765625" disk.nvme0c0n1.in‚ 409.17578125$ memory_percent‚94.20979022979736* proc.memory.percent‚17.531394958496094 cpu‚25.319151227895507 disk.sda2.in‚0 disk.dm-1.out‚ 172.4765625Û{C¬’Ñ‚(:‡ Æ¢åÔª°Û gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚92.937' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ¢î&Ò‚(:› Æ¢åÔ‡ñ€‘ network.recv‚ 1.624674e+06! disk.dm-0.out‚25251.736328125 disk.sda1.in‚0 disk.drbd1777.in‚0& disk.nvme0c0n1.out‚25251.736328125 disk.rbd0.out‚0 disk.sda.in‚0) disk./.usagePercent‚73.87525982716664 disk.rbd2.out‚ 132.203125 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.531394958496094 disk.drbd1777.out‚0 disk.rbd3.in‚0$ memory_percent‚94.20828819274902' proc.memory.availableMB‚ 948.9140625 proc.memory.rssMB‚ 2872.34375 cpu‚25.318231648819253 disk.rbd3.out‚0% disk./.usageGB‚1100.7947578430176 disk.dm-1.out‚ 173.04296875$ disk.nvme0n1.out‚25251.736328125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 173.04296875 disk.dm-0.in‚ 409.17578125 network.sent‚ 1.3117658e+07 disk.rbd1.in‚0 disk.rbd1.out‚ 2.294921875" disk.nvme0c0n1.in‚ 409.17578125 disk.sda2.out‚0 disk.nvme0n1.in‚ 409.17578125 disk.sda2.in‚0 disk.sda.out‚ 173.04296875]øî?2Ó- _wandbruntime‚4401  _runtime‚4401vºõÀÉÔà ' lr-AdamW/pg1‚0.00013926769291120735 ' lr-AdamW/pg2‚0.00013926769291120735 trainer/global_step‚5849 % _timestamp‚1.7884327169561398e+09 _runtime‚4401.660884692 _step‚476¿ˆ©fÎÕÈ trainer/global_step‚5849 % _timestamp‚1.7884327169561398e+09 _runtime‚4401.660884692 _step‚476 ' lr-AdamW/pg1‚0.00013926769291120735 ' lr-AdamW/pg2‚0.00013926769291120735ÜÁaEà2Ö- _wandbruntime‚4401  _runtime‚4401%q1à'×ÚO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343850507247104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84724308 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194934698 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77297813 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375350565235712 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚338963357004288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49427508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4869835057168 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84724310 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4887824608256 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110190350 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5128591019520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110210388 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27870748 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343851181612544 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343850507247104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4887168245760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330098874555392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 194914658 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84724310 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4869180619412 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚330098891594752 _step‚477 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚338963339001344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334968726651920 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚5849 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77298258 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 §ÒKå'ØßO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49427508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4869835057168 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84724310 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4887824608256 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110190350 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5128591019520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110210388 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27870748 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343851181612544 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343850507247104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4887168245760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330098874555392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 194914658 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84724310 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4869180619412 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚330098891594752 _step‚477 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚338963339001344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334968726651920 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚5849 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77298258 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343850507247104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84724308 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194934698 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77297813 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375350565235712 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚338963357004288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128ÝTÑhm2Ù- _wandbruntime‚4402  _runtime‚4402i¹>(ÚäM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339019692040192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370283782286336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194967194 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330153778685952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4870651492340 trainer/global_step‚5849 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_byte“çQ6Âs.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84738324 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84738326 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77311012 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49435676 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194967194 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4888644049408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343907679713792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚339019710043136 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77310558 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49435678 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330153795725312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _runtime‚4402.318154035 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888644049408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110208804 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4887987673600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4887987673600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5129451159552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335023775740328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194947128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339019710043136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375413065640960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5129283354624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788432717612765e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375413257385984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343908354092544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27874882 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚335024447217652 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110208804 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808ßIï&ÛéM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888644049408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110208804 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4887987673600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4887987673600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5129451159552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335023775740328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194947128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339019710043136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375413065640960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5129283354624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788432717612765e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375413257385984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343908354092544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27874882 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚335024447217652 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110208804 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339019692040192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370283782286336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194967194 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330153778685952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4870651492340 trainer/global_step‚5849 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84738324 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84738326 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77311012 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49435676 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194967194 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4888644049408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343907679713792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚339019710043136 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77310558 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49435678 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330153795725312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _runtime‚4402.318154035ÞÈHò2Ü- _wandbruntime‚4402  _runtime‚4402ç&P<[ÝÕ 0loss/FRt1/assignment_loss‚1.7783973283237882 2loss/SRqqt4/assignment_loss‚1.5106076207011938 /loss/FBt3/detection_loss‚2.8339674364436758 epoch‚4 _runtime‚4402.319606531 /loss/FRt2/assignment_loss‚2.534716396830803 1loss/SRqqt2/assignment_loss‚1.429589629100234 1loss/FBt1/assignment_loss‚0.16391134856290138 1loss/SRqqt2/detection_loss‚1.7913830633841308 0loss/FRt4/assignment_loss‚3.6184848099946976 2loss/SRqqt1/assignment_loss‚1.0759138534365988 /loss/FRt1/detection_loss‚1.8298210567070379 .loss/FRt2/detection_loss‚2.465930072374122 .loss/FRt3/detection_loss‚2.801461538299918 1loss/SRqqt4/detection_loss‚2.0267377209506536 %loss/total_loss‚42.97737761510521 _step‚479 0loss/FBt2/assignment_loss‚0.3392896570217999 /loss/FBt4/detection_loss‚2.0725230475266776 /loss/FRt3/assignment_loss‚3.226563733071089 1loss/FBt3/assignment_loss‚0.19461982991461726 .loss/FRt4/detection_loss‚2.344112646455566 trainer/global_step‚5849 1loss/SRqqt3/assignment_loss‚2.101107011316344 1loss/FBt4/assignment_loss‚0.37792708127138513 1loss/SRqqt1/detection_loss‚1.0818842630833387 /loss/FBt1/detection_loss‚0.6589586045505927 % _timestamp‚1.7884327176146786e+09 1loss/SRqqt3/detection_loss‚2.6580158676952124 /loss/FBt2/detection_loss‚2.0614539980888367?Bîå`ÞÚ 0loss/FRt4/assignment_loss‚3.6184848099946976 2loss/SRqqt1/assignment_loss‚1.0759138534365988 /loss/FRt1/detection_loss‚1.8298210567070379 .loss/FRt2/detection_loss‚2.465930072374122 .loss/FRt3/detection_loss‚2.801461538299918 1loss/SRqqt4/detection_loss‚2.0267377209506536 %loss/total_loss‚42.97737761510521 _step‚479 0loss/FBt2/assignment_loss‚0.3392896570217999 /loss/FBt4/detection_loss‚2.0725230475266776 /loss/FRt3/assignment_loss‚3.226563733071089 1loss/FBt3/assignment_loss‚0.19461982991461726 .loss/FRt4/detection_loss‚2.344112646455566 trainer/global_step‚5849 1loss/SRqqt3/assignment_loss‚2.101107011316344 1loss/FBt4/assignment_loss‚0.37792708127138513 1loss/SRqqt1/detection_loss‚1.0818842630833387 /loss/FBt1/detection_loss‚0.6589586045505927 % _timestamp‚1.7884327176146786e+09 1loss/SRqqt3/detection_loss‚2.6580158676952124 /loss/FBt2/detection_loss‚2.0614539980888367 0loss/FRt1/assignment_loss‚1.7783973283237882 2loss/SRqqt4/assignment_loss‚1.5106076207011938 /loss/FBt3/detection_loss‚2.8339674364436758 epoch‚4 _runtime‚4402.319606531 /loss/FRt2/assignment_loss‚2.534716396830803 1loss/SRqqt2/assignment_loss‚1.429589629100234 1loss/FBt1/assignment_loss‚0.16391134856290138 1loss/SRqqt2/detection_loss‚1.7913830633841308ßàïIà“ß‚(:ˆ Õ¢åÔþ¡ÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚105.443' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EŠ+ë à‚(:• Õ¢åÔ¨ìœ disk.rbd0.in‚0 proc.memory.rssMB‚ 2872.34375 disk.rbd3.out‚0! disk.dm-0.out‚25350.919921875 disk.sda.in‚0 disk.sda3.out‚ 173.59375$ memory_percent‚94.21007633209229( proc.memory.availableMB‚ 948.62109375 cpu‚25.317277426236465 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 409.17578125$ disk.nvme0n1.out‚25350.919921875 disk.rbd1.out‚ 2.3271484375 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚25350.919921875 disk.sda2.out‚0) disk./.usagePercent‚73.87525368298583% disk./.usageGB‚1100.7946662902832 disk.dm-1.out‚ 173.59375 disk.rbd2.in‚0* proc.memory.percent‚17.531394958496094 network.sent‚ 1.312392e+07 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 409.17578125 disk.sda3.in‚0 network.recv‚ 1.629112e+06 disk.rbd1.in‚0 disk.rbd2.out‚ 133.79296875 disk.sda.out‚ 173.59375 disk.nvme0n1.in‚ 409.17578125åȆ“á‚(:ˆ ä¢åÔö£Ž gpu.0.gpu‚26.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚155.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.0442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y…–8&â‚(:› ä¢åÔœ÷ù disk.rbd1.out‚ 2.3271484375 disk.sda3.out‚ 174.0859375 disk.rbd3.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 415.48828125 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0$ memory_percent‚94.20855045318604& disk.nvme0c0n1.out‚25450.779296875 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 2872.34375 proc.cpu.threads‚17 network.recv‚ 1.635465e+06$ disk.nvme0n1.out‚25450.779296875 disk.rbd2.in‚0 disk.rbd2.out‚ 133.79296875 disk.sda.out‚ 174.0859375 cpu‚25.316379938000114 disk.sda2.in‚0 disk.dm-0.in‚ 415.48828125 disk.sda2.out‚0 network.sent‚ 1.3232065e+07$ disk./.usageGB‚1100.794189453125 disk.dm-1.out‚ 174.0859375 disk.drbd1777.in‚0! disk.dm-0.out‚25450.779296875( proc.memory.availableMB‚ 948.87109375* proc.memory.percent‚17.531394958496094) disk./.usagePercent‚73.87522168204404 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 415.48828125 disk.rbd0.out‚0‹xLb2ã- _wandbruntime‚4436  _runtime‚4436b¿üFÉäà ' lr-AdamW/pg2‚0.00013767275998248876 trainer/global_step‚5899 % _timestamp‚1.7884327518590565e+09 _runtime‚4436.563797651 _step‚480 ' lr-AdamW/pg1‚0.00013767275998248876×‹Ñ ÎåÈ ' lr-AdamW/pg1‚0.00013767275998248876 ' lr-AdamW/pg2‚0.00013767275998248876 trainer/global_step‚5899 % _timestamp‚1.7884327518590565e+09 _runtime‚4436.563797651 _step‚480àáç½2æ- _wandbruntime‚4436  _runtime‚44361L¦¯™!çÛO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4910015958820 trainer/global_step‚5899 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85425108 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77935322 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341781008946688 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4928810414080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196541510 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5171891089920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4928154051584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4928810414080 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚332844098122752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚346709819360768 % _timestamp‚1.7884327518597913e+09 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚481 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341780990943744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚4436.564793337 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49835910 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49835908 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346709144995328 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111116402 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚373304668431872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111116402 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚346709819360768 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111136440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_baqsÆ÷Htch_start/active.large_pool.allocated‚85425110 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341781008946688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373304692371968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28099414 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚346709144995328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378476393800704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚00§æ'èàO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28099414 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚346709144995328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378476393800704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4910015958820 trainer/global_step‚5899 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85425108 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77935322 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341781008946688 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4928810414080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196541510 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5171891089920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4928154051584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4928810414080 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚332844098122752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚346709819360768 % _timestamp‚1.7884327518597913e+09 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚481 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341780990943744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚4436.564793337 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49835910 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49835908 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346709144995328 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111116402 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚373304668431872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111116402 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚346709819360768 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111136440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85425110 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341781008946688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373304692371968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0á"F 2é- _wandbruntime‚4437  _runtime‚4437‘≿ë&êåM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378539095044096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚337809817607736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341837361985536 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196594046 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49844076 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5172751934464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4911486831748 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4929629855232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5172584129536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚346766317462016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚4437.225328766 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111134856 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77947977 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28103901 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346766991840768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332899002253312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928973479424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚373366319169536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85439124 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111154920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337810489085060 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85439126 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332898985213952 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341837343982592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341837361985536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4928973479424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196594046 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 _step‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111134856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49844078 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85439126 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚346766991840768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85439124 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77947977 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28103901 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346766991840768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332899002253312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928973479424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚373366319169536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85439124 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111154920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337810489085060 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85439126 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332898985213952 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341837343982592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341837361985536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4928973479424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196594046 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 _step‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111134856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49844078 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85439126 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚346766991840768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85439124 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378539095044096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚337809817607736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341837361985536 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196594046 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49844076 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5172751934464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4911486831748 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4929629855232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5172584129536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚346766317462016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚4437.225328766 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111134856â¾ß¡Ë2ì- _wandbruntime‚4437  _runtime‚4437¤áOZbíØ 0loss/FRt1/assignment_loss‚1.3135856756797204 0loss/FBt2/assignment_loss‚0.2268718097184319 1loss/FBt4/assignment_loss‚0.13473725505173206 1loss/SRqqt2/detection_loss‚1.9065459243332346 0loss/FRt3/assignment_loss‚3.8334200263023375 2loss/SRqqt1/assignment_loss‚0.8363273258986218 1loss/SRqqt1/detection_loss‚0.9991994094635759 _runtime‚4437.226349859 epoch‚4 trainer/global_step‚5899 0loss/FRt2/assignment_loss‚2.5238278457096643 1loss/SRqqt2/assignment_loss‚1.252416992104069 1loss/FBt1/assignment_loss‚0.17519360121193048 /loss/FBt2/detection_loss‚3.5985326766967773 _step‚483 /loss/FRt3/detection_loss‚2.5112280145287516 /loss/FBt1/detection_loss‚0.6596143321924804 .loss/FBt3/detection_loss‚2.559244236775807 2loss/SRqqt4/assignment_loss‚1.3894132983927825 1loss/FBt3/assig‘ ?¬ünment_loss‚0.19766225983455246 1loss/SRqqt4/detection_loss‚1.9402037725156667 0loss/FRt4/assignment_loss‚3.5322311433653035 2loss/SRqqt3/assignment_loss‚2.0862194710859545 .loss/FRt4/detection_loss‚2.268020727982124 &loss/total_loss‚44.231773947591634 % _timestamp‚1.7884327525217104e+09 .loss/FRt2/detection_loss‚2.492189668757575 /loss/FRt1/detection_loss‚1.9774161622119255 0loss/SRqqt3/detection_loss‚2.619113480603253 /loss/FBt4/detection_loss‚3.1985588371753693¸ ØÓcîÝ 1loss/SRqqt4/detection_loss‚1.9402037725156667 0loss/FRt4/assignment_loss‚3.5322311433653035 2loss/SRqqt3/assignment_loss‚2.0862194710859545 .loss/FRt4/detection_loss‚2.268020727982124 &loss/total_loss‚44.231773947591634 % _timestamp‚1.7884327525217104e+09 .loss/FRt2/detection_loss‚2.492189668757575 /loss/FRt1/detection_loss‚1.9774161622119255 0loss/SRqqt3/detection_loss‚2.619113480603253 /loss/FBt4/detection_loss‚3.1985588371753693 0loss/FRt1/assignment_loss‚1.3135856756797204 0loss/FBt2/assignment_loss‚0.2268718097184319 1loss/FBt4/assignment_loss‚0.13473725505173206 1loss/SRqqt2/detection_loss‚1.9065459243332346 0loss/FRt3/assignment_loss‚3.8334200263023375 2loss/SRqqt1/assignment_loss‚0.8363273258986218 1loss/SRqqt1/detection_loss‚0.9991994094635759 _runtime‚4437.226349859 epoch‚4 trainer/global_step‚5899 0loss/FRt2/assignment_loss‚2.5238278457096643 1loss/SRqqt2/assignment_loss‚1.252416992104069 1loss/FBt1/assignment_loss‚0.17519360121193048 /loss/FBt2/detection_loss‚3.5985326766967773 _step‚483 /loss/FRt3/detection_loss‚2.5112280145287516 /loss/FBt1/detection_loss‚0.6596143321924804 .loss/FBt3/detection_loss‚2.559244236775807 2loss/SRqqt4/assignment_loss‚1.3894132983927825 1loss/FBt3/assignment_loss‚0.19766225983455246ãt¦bZ“ï‚(:ˆ ó¢åÔ¹«ç gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚102.381' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4762 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥Á 0ð‚(:¥ ó¢åÔÚšÏ network.recv‚ 1.639949e+06% disk./.usageGB‚1100.7941932678223 disk.sda.out‚ 174.68359375 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 415.48828125 disk.rbd2.out‚ 133.87109375 cpu‚25.315477466573956 disk.sda.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0# disk.dm-0.out‚25451.23291015625* proc.memory.percent‚17.531394958496094 disk.dm-1.in‚0 disk.nvme0n1.in‚ 415.48828125 disk.sda2.in‚0 disk.sda3.out‚ 174.68359375 disk.drbd1777.in‚0( proc.memory.availableMB‚ 948.40234375 proc.cpu.threads‚17 network.sent‚ 1.323833e+07 disk.rbd1.out‚ 2.39013671875( disk.nvme0c0n1.out‚25451.23291015625 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 proc.memory.rssMB‚ 2872.34375$ memory_percent‚94.21141147613525 disk.dm-1.out‚ 174.68359375 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 415.48828125) disk./.usagePercent‚73.87522193805158& disk.nvme0n1.out‚25451.23291015625×v¢W“ñ‚(:ˆ ‚£åÔü€Ý gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fߺÎ+ò‚(:  ‚£åÔ´»ý) disk./.usagePercent‚73.87522987428514# disk.dm-0.out‚25551.41162109375 disk.sda.out‚ 175.1171875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚25551.41162109375& proc.memory.availableMB‚ 949.046875 disk.drbd1777.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 431.81640625% disk./.usageGB‚1100.7943115234375& disk.nvme0n1.out‚25551.41162109375 disk.rbd1.out‚ 2.39013671875 disk.rbd0.in‚0 network.sent‚ 1.3346188e+07 network.recv‚ 1.645561e+06 disk.dm-1.out‚ 175.1171875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 431.81640625 proc.memory.rssMB‚ 2872.34375* proc.memory.percent‚17.531394958496094 cpu‚25.31459620782225 disk.sda3.in‚0 disk.sda3.out‚ 175.1171875 disk.dm-0.in‚ 431.81640625 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd2.out‚ 133.87109375 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0$ memory_percent‚94.20747756958008Üá5Ê‘ó‚(:† ‘£åÔþåÕ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚104.04' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.808 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É0p-ô‚(:¢ ‘£åÔóÿ×& disk.nvme0n1.out‚25650.49365234375 disk.rbd0.in‚0! disk.rbd2.out‚133.97314453125 disk.drbd1777.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 948.84765625 network.sent‚ 1.3352408e+07 network.recv‚ 1.649851e+06) disk./.usagePercent‚73.87522603417213 disk.dm-0.in‚ 431.81640625 disk.nvme0n1.in‚ 431.81640625 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚25650.49365234375# memory_percent‚94.2086935043335 proc.memory.rssMB‚ 2872.34375 disk.sda.out‚ 175.71484375 disk.dm-1.out‚ 175.71484375 disk.rbd2.in‚0* proc.memory.percent‚17.531394958496094# disk.dm-0.out‚25650.49365234375 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 cpu‚25.31366133214802 disk.rbd1.out‚2.453125% disk./.usageGB‚1100.7942543029785 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 431.81640625 disk.sda3.in‚0 disk.sda3.out‚ 175.71484375 proc.cpu.threads‚17‘¸tf2õ- _wandbruntime‚4471  _runtime‚4471T°¤Èö ' lr-AdamW/pg2‚0.00013607275018356324 trainer/global_step‚5949 $ _timestamp‚1.788432786903206e+09 _runtime‚4471.607983543 _step‚484 ' lr-AdamW/pg1‚0.00013607275018356324OòýÍ÷Ç ' lr-AdamW/pg1‚0.00013607275018356324 ' lr-AdamW/pg2‚0.00013607275018356324 trainer/global_step‚5949 $ _timestamp‚1.788432786903206e+09 _runtime‚4471.607983543 _step‚484äL82ø- _wandbruntime‚4471  _runtime‚4471éÅWá'ùÛO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198168362 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4969139857408 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349567782743552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340540138909620 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4950851298228 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚381602003495424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86125910 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198188402 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28328978 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚335589304650752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚381602184768000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376387205315072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5949 _step‚485 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335589287611392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86125908 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349568457108992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚344598642886144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340540810386736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86125910 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112062492 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78573737 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4969796219904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78573737 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4969796219904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349567782743552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340540138909620 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872å'd2û- _wandbruntime‚4472  _runtime‚4472DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884327875562701e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198220898 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349625629588992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4969959285248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86139924 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28333975 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381664497229312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚335644208781312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4970615661056 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5949 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5215641176576 ZFDeviceStatsMonitor.on_train_¦m¥äëbatch_end/requested_bytes.large_pool.freed‚335644191741952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198200832 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112080972 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340596530952468 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4951667733192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86139926 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198200832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86139926 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78585990 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28333514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112060908 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344655013927936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198220898 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50252476 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414‰÷“ýð&ýêM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349625629588992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4969959285248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86139924 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28333975 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381664497229312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚335644208781312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4970615661056 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5949 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5215641176576 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚335644191741952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198200832 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112080972 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340596530952468 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4951667733192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86139926 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198200832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86139926 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78585990 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28333514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112060908 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344655013927936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198220898 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50252476 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349625629588992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78586453 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚340595859475144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112060908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884327875562701e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198220898 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0拾ª»2þ- _wandbruntime‚4472  _runtime‚4472-dTÿÎ /loss/FRt4/detection_loss‚2.3700407198289546 0loss/FRt1/assignment_loss‚1.3517951369285583 /loss/FBt3/assignment_loss‚0.226462224396073 0loss/FBt4/assignment_loss‚0.2608916673978621 0loss/FBt1/detection_loss‚0.44898991605221483 _runtime‚4472.262689356 0loss/FRt2/assignment_loss‚2.6988251114433464 .loss/FBt3/detection_loss‚2.859817035496235 /loss/FBt4/detection_loss‚1.8768796649846164 _step‚487 /loss/FRt2/detection_loss‚2.5352305844426155 #loss/total_loss‚44.314069002097 epoch‚4 .loss/FRt4/assignment_loss‚4.14618339189669 1loss/SRqqt2/detection_loss‚1.9361498510198933 /loss/SRqqt4/detection_loss‚2.19267338514328 .loss/FBt2/detection_loss‚3.602315679192543 trainer/global_step‚5949 /loss/FRt3/assignment_loss‚3.470349137599652 1loss/FBt1/assignment_loss‚0.14034903546324998 1loss/SRqqt1/detection_loss‚0.9595474987808201 $ _timestamp‚1.788432787558048e+09 /loss/FRt1/detection_loss‚1.7476949345033903 /loss/SRqqt3/detection_loss‚2.21319705758776 2loss/SRqqt1/assignment_loss‚1.2272757231257856 2loss/SRqqt2/assignment_loss‚1.1863164332296168 2loss/SRqqt3/assignment_loss‚1.7431007625801223 2loss/SRqqt4/assignment_loss‚1.5101123976831634 0loss/FBt2/assignment_loss‚0.2805602343287319 /loss/FRt3/detection_loss‚3.3293114189918223ããÒëY€Ó 2loss/SRqqt2/assignment_loss‚1.1863164332296168 2loss/SRqqt3/assignment_loss‚1.7431007625801223 2loss/SRqqt4/assignment_loss‚1.5101123976831634 0loss/FBt2/assignment_loss‚0.2805602343287319 /loss/FRt3/detection_loss‚3.3293114189918223 /loss/FRt4/detection_loss‚2.3700407198289546 0loss/FRt1/assignment_loss‚1.3517951369285583 /loss/FBt3/assignment_loss‚0.226462224396073 0loss/FBt4/assignment_loss‚0.2608916673978621 0loss/FBt1/detection_loss‚0.44898991605221483 _runtime‚4472.262689356 0loss/FRt2/assignment_loss‚2.6988251114433464 .loss/FBt3/detection_loss‚2.859817035496235 /loss/FBt4/detection_loss‚1.8768796649846164 _step‚487 /loss/FRt2/detection_loss‚2.5352305844426155 #loss/total_loss‚44.314069002097 epoch‚4 .loss/FRt4/assignment_loss‚4.14618339189669 1loss/SRqqt2/detection_loss‚1.9361498510198933 /loss/SRqqt4/detection_loss‚2.19267338514328 .loss/FBt2/detection_loss‚3.602315679192543 trainer/global_step‚5949 /loss/FRt3/assignment_loss‚3.470349137599652 1loss/FBt1/assignment_loss‚0.14034903546324998 1loss/SRqqt1/detection_loss‚0.9595474987808201 $ _timestamp‚1.788432787558048e+09 /loss/FRt1/detection_loss‚1.7476949345033903 /loss/SRqqt3/detection_loss‚2.21319705758776 2loss/SRqqt1/assignment_loss‚1.2272757231257856ç1Pœ‚(:‘  £åÔ½åÚ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚90.65' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.130000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O˜¢™+‚‚(:   £åÔ…üþ% disk./.usageGB‚1100.7943115234375 disk.rbd1.out‚2.453125 proc.memory.rssMB‚ 2872.34375 cpu‚25.312769412574735 network.sent‚ 1.335791e+07 disk.rbd0.out‚0 disk.drbd1777.out‚0# disk.dm-0.out‚25749.93994140625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0! disk.rbd2.out‚133.97314453125 disk.nvme0n1.in‚ 431.81640625 disk.sda1.in‚0 disk.dm-1.out‚ 176.0859375( disk.nvme0c0n1.out‚25749.93994140625 disk.rbd3.out‚0( proc.memory.availableMB‚ 948.82421875* proc.memory.percent‚17.531394958496094 disk.sda.in‚0 disk.rbd1.in‚0$ memory_percent‚94.20883655548096 proc.cpu.threads‚17 network.recv‚ 1.654096e+06) disk./.usagePercent‚73.87522987428514 disk.sda.out‚ 176.0859375 disk.sda3.in‚0 disk.sda3.out‚ 176.0859375 disk.rbd3.in‚0 disk.dm-0.in‚ 431.81640625 disk.rbd2.in‚0 disk.drbd1777.in‚0& disk.nvme0n1.out‚25749.93994140625 disk.sda1.out‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 431.81640625Gö›S’ƒ‚(:‡ ¯£åÔ”ð gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚95.731' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Áÿ\-„‚(:¢ ¯£åԜԽ disk.sda2.out‚0 disk.rbd0.in‚0! disk.rbd2.out‚135.56298828125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 431.81640625 disk.nvme0n1.in‚ 431.81640625 disk.sda3.in‚0# disk.dm-0.out‚25849.11083984375$ memory_percent‚94.21148300170898 proc.memory.rssMB‚ 2872.34375 disk.sda1.in‚0 disk.dm-1.out‚ 176.78515625" disk.nvme0c0n1.in‚ 431.81640625 disk.sda3.out‚ 176.78515625& proc.memory.availableMB‚ 948.390625 proc.cpu.threads‚17 network.sent‚ 1.3466499e+07) disk./.usagePercent‚73.87522321808926 disk.rbd1.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 176.78515625( disk.nvme0c0n1.out‚25849.11083984375 disk.rbd0.out‚0 disk.drbd1777.in‚0 cpu‚25.311904324045326 disk.dm-1.in‚0 disk.rbd1.out‚2.453125* proc.memory.percent‚17.531394958496094 disk.sda1.out‚0& disk.nvme0n1.out‚25849.11083984375 disk.rbd2.in‚0% disk./.usageGB‚1100.7942123413086 disk.sda.in‚0 network.recv‚ 1.660043e+06 disk.sda2.in‚0\¹2…- _wandbruntime‚4506  _runtime‚4506ô¹EóȆ ' lr-AdamW/pg1‚0.00013446799695407465 ' lr-AdamW/pg2‚0.00013446799695407465 trainer/global_step‚5999 $ _timestamp‚1.788432821732474e+09 _runtime‚4506.437217662 _step‚488H°eÍ‡Ç ' lr-AdamW/pg1‚0.00013446799695407465 ' lr-AdamW/pg2‚0.00013446799695407465 trainer/global_step‚5999 $ _timestamp‚1.788432821732474e+09 _runtime‚4506.437217662 _step‚488èÞM[l2ˆ- _wandbruntime‚4506  _runtime‚4506ÚRá'‰ÛO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 % _timestamp‚1.7884328217332058e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚352427094857216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5010125663232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4992341075392 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _step‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50652710 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199815254 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚347416294828544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚343326180777028 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79210497 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86826710 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚352426420491776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199795214 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347416312831488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28557352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343326852254144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338334494139392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5010782025728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347416312831488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112968506 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347416294828544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4991686637636 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338334511178752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352426420491776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚379469766138368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚4506.438164986 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86826708 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5010125663232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5257935678464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28557787 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384727518447104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379469742198272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112968506 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚352427094857216 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86826710 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5257776248832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79210060 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86826708 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112988544 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338334511178752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batÓFwi!ch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352426420491776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚379469766138368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚4506.438164986 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86826708 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5010125663232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5257935678464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28557787 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384727518447104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379469742198272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112968506 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚352427094857216 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86826710 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5257776248832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79210060 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86826708 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112988544 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199815254 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚347416294828544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚343326180777028 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79210497 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86826710 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚352426420491776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199795214 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347416312831488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28557352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343326852254144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338334494139392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5010782025728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347416312831488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112968506éz1¢42‹- _wandbruntime‚4507  _runtime‚4507|:’"ë&ŒåM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86840726 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86840724 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚379531416876032 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4993157510564 _runtime‚4507.107514272 _step‚490 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199827684 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚379531392935936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347472665870336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347472647867392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4992503072600 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚352484267337216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347472665870336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5999 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86840724 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5258806608384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚352483592958464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5010945091072 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚338389398269952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113007024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112986960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86840726 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50660876 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 170773504 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352484267337216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347472647867392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5011601466880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384790052710912 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚343382572819876 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚343381901342552 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199847750 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79223075 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113007024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384790223484416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 146833408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199847750 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79222647 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28562197 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112986960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5258659774976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5010945091072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚352484267337216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347472665870336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5999 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86840724 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5258806608384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚352483592958464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5010945091072 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚338389398269952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113007024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112986960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86840726 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50660876 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 170773504 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352484267337216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347472647867392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5011601466880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384790052710912 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚343382572819876 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚343381901342552 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199847750 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79223075 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113007024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384790223484416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 146833408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199847750 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79222647 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28562197 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112986960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5258659774976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5010945091072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096êæ¨ ë2Ž- _wandbruntime‚4507  _runtime‚4507¿2ëŽZÔ _runtime‚ 4507.10868123 2loss/SRqqt3/assignment_loss‚1.8853093030698158 1loss/SRqqt1/detection_loss‚1.1149057301720886 0loss/FRt4/assignment_loss‚3.1505499303340914 2loss/SRqqt4/assignment_loss‚1.4518815837800503 0loss/FRt1/assignment_loss‚1.6497726124875687 0loss/FBt2/assignment_loss‚0.6518837373487116 0loss/SRqqt3/detection_loss‚2.435917643063209 /loss/FBt3/detection_loss‚2.8094777345657347 1loss/FBt4/assignment_loss‚0.34302303178066557 ,loss/FRt1/detection_loss‚1.6576322444952 1loss/SRqqt4/detection_loss‚2.1049553396853993 .loss/FBt1/detection_loss‚0.724515103836222 _step‚491 2loss/SRqqt1/assignment_loss‚1.0762534814138869 1loss/FBt1/assignment_loss‚0.18644129047138222 .loss/FRt4/detection_loss‚2.260048888921738 %loss/total_loss‚42.71011001512617 trainer/global_step‚5999 % _timestamp‚1.7884328224040394e+09 0loss/FRt2/assignment_loss‚3.1827248019330643 2loss/SRqqt2/assignment_loss‚1.2846878481962318 /loss/FRt2/detection_loss‚2.3042128360738943 /loss/FRt3/detection_loss‚2.5265572521421644 1loss/SRqqt2/detection_loss‚1.6821957150740283 /loss/FBt2/detection_loss‚2.8400353491306305 0loss/FRt3/assignment_loss‚2.7948725912306043 1loss/FBt3/assignment_loss‚0.14058373701991514 .loss/FBt4/detection_loss‚2.451672228899869 epoch‚4 ûA_Ù trainer/global_step‚5999 % _timestamp‚1.7884328224040394e+09 0loss/FRt2/assignment_loss‚3.1827248019330643 2loss/SRqqt2/assignment_loss‚1.2846878481962318 /loss/FRt2/detection_loss‚2.3042128360738943 /loss/FRt3/detection_loss‚2.5265572521421644 1loss/SRqqt2/detection_loss‚1.6821957150740283 /loss/FBt2/detection_loss‚2.8400353491306305 0loss/FRt3/assignment_loss‚2.7948725912306043 1loss/FBt3/assignment_loss‚0.14058373701991514 .loss/FBt4/detection_loss‚2.451672228899869 epoch‚4 _runtime‚ 4507.10868123 2loss/SRqqt3/assignment_loss‚1.8853093030698158 1loss/SRqqt1/detection_loss‚1.1149057301720886 0loss/FRt4/assignment_loss‚3.1505499303340914 2loss/SRqqt4/assignment_loss‚1.4518815837800503 0loss/FRt1/assignment_loss‚1.6497726124875687 0loss/FBt2/assignment_loss‚0.6518837373487116 0loss/SRqqt3/detection_loss‚2.435917643063209 /loss/FBt3/detection_loss‚2.8094777345657347 1loss/FBt4/assignment_loss‚0.34302303178066557 ,loss/FRt1/detection_loss‚1.6576322444952 1loss/SRqqt4/detection_loss‚2.1049553396853993 .loss/FBt1/detection_loss‚0.724515103836222 _step‚491 2loss/SRqqt1/assignment_loss‚1.0762534814138869 1loss/FBt1/assignment_loss‚0.18644129047138222 .loss/FRt4/detection_loss‚2.260048888921738 %loss/total_loss‚42.71011001512617ëˆÚñxž‘‚(:“ ¾£åÔÉ„ƒŽ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚102.499' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.499799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œá'_Â’‚(:’ ¾£åÔ뽩‘ network.sent‚ 1.3471743e+07 network.recv‚ 1.663756e+06 disk.sda.out‚ 177.390625 disk.sda3.in‚0 disk.rbd3.in‚0# disk.nvme0n1.out‚25849.73828125 disk.sda2.out‚0 disk.rbd0.out‚0$ memory_percent‚94.21017169952393 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.dm-0.out‚25849.73828125 proc.memory.rssMB‚ 2872.34375% disk./.usageGB‚1100.7942695617676 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.out‚2.453125% disk.nvme0c0n1.out‚25849.73828125 disk.sda3.out‚ 177.390625* proc.memory.percent‚17.531394958496094 disk.dm-1.in‚0 disk.nvme0n1.in‚ 431.81640625 disk.sda1.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 431.81640625! disk.rbd2.out‚135.56298828125 disk.dm-0.in‚ 431.81640625( proc.memory.availableMB‚ 948.60546875 disk.dm-1.out‚ 177.390625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 cpu‚25.310974521954) disk./.usageP¶ï£[ercent‚73.87522705820227 disk.sda2.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17Ì:-ž“‚(:“ Í£åÔ£ºÍ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚133.574' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.714800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%n(:‰ Í£åÔÍêú network.sent‚ 1.358064e+07% disk.nvme0c0n1.out‚25948.87890625 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 431.83984375( proc.memory.availableMB‚ 948.33984375* proc.memory.percent‚17.531394958496094) disk./.usagePercent‚73.87522321808926 proc.memory.rssMB‚ 2872.34375 disk.drbd1777.in‚0 disk.drbd1777.out‚0% disk./.usageGB‚1100.7942123413086 disk.sda.out‚177.875 disk.nvme0n1.in‚ 431.83984375# disk.nvme0n1.out‚25948.87890625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚177.875 disk.rbd1.out‚2.453125 disk.sda2.out‚0 disk.rbd2.out‚135.7275390625 disk.sda3.out‚177.875$ memory_percent‚94.21179294586182 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 431.83984375 disk.dm-0.out‚25948.87890625 cpu‚25.31011953700858 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 1.670241e+06ÕýïB’•‚(:‡ Ü£åÔ¯¯•Ž gpu.0.gpu‚59.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.593' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.5186 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Br¹–‚(:’ Ü£åÔéÈ¿’ network.recv‚ 1.674398e+06 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 178.35546875 disk.rbd2.out‚135.7275390625 disk.dm-0.out‚ 26047.953125 disk.sda.out‚ 178.35546875 disk.dm-1.in‚0 cpu‚25.31635816072636) disk./.usagePercent‚73.87520324950158 disk.nvme0n1.in‚ 431.83984375! disk.nvme0n1.out‚ 26047.953125" disk.nvme0c0n1.in‚ 431.83984375 disk.sda1.in‚0% proc.memory.availableMB‚ 947.59375 network.sent‚ 1.3586098e+07 disk.sda2.out‚0 disk.rbd0.in‚0$ memory_percent‚94.21634674072266 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 431.83984375 disk.rbd0.out‚0% disk./.usageGB‚1100.7939147949219 disk.rbd1.out‚2.453125# disk.nvme0c0n1.out‚ 26047.953125 disk.sda3.out‚ 178.35546875 disk.drbd1777.out‚0 disk.sda1.out‚0* proc.memory.percent‚17.537927627563477" proc.memory.rssMB‚ 2873.4140625 proc.cpu.threads‚17ëwE6“—‚(:ˆ ë£åÔÆ‹ƒŽ gpu.0.gpu‚17.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚158.136' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.6272 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}¾¼-˜‚(:¢ ë£åÔκç disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0$ memory_percent‚94.21741962432861% disk.nvme0n1.out‚26147.0947265625 disk.rbd1.in‚0 disk.sda3.out‚ 179.09765625 disk.drbd1777.out‚0* proc.memory.percent‚17.537927627563477 proc.cpu.threads‚17' disk.nvme0c0n1.out‚26147.0947265625 disk.rbd2.out‚135.8056640625 disk.dm-1.out‚ 179.09765625( proc.memory.availableMB‚ 947.41796875 network.recv‚ 1.678573e+06" disk.nvme0c0n1.in‚ 431.83984375 disk.rbd3.in‚0" disk.dm-0.out‚26147.0947265625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 179.09765625 disk.dm-1.in‚0) disk./.usagePercent‚73.87520606558446% disk./.usageGB‚1100.7939567565918 disk.nvme0n1.in‚ 431.83984375 disk.rbd1.out‚2.453125 cpu‚25.330661109834683 network.sent‚ 1.3592405e+07 disk.rbd3.out‚0 disk.dm-0.in‚ 431.83984375" proc.memory.rssMB‚ 2873.4140625¾t™™‚(:’ ú£åÔÉÛ× gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚164.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I‚sN,š‚(:¡ ú£åÔßõ‘‘& disk.nvme0n1.out‚26147.75439453125 disk.rbd3.in‚0* proc.memory.percent‚17.097091674804688 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 179.55859375 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 1.3596601e+07" disk.nvme0c0n1.in‚ 431.83984375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 179.55859375$ memory_percent‚93.77298355102539 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 431.83984375 disk.sda2.in‚0 disk.rbd2.out‚137.3173828125 disk.sda3.in‚0 disk.drbd1777.in‚0' proc.memory.availableMB‚ 1020.234375 proc.memory.rssMB‚ 2801.1875 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 179.55859375 disk.dm-0.in‚ 431.83984375 cpu‚25.346152025890454# disk.dm-0.out‚26147.75439453125 network.recv‚ 1.681478e+06 disk.rbd1.out‚2.453125( disk.nvme0c0n1.out‚26147.75439453125) disk./.usagePercent‚73.87523985857898% disk./.usageGB‚1100.7944602966309Ùyöe2›- _wandbruntime‚4585  _runtime‚4585¿£RG*œÁT YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚356879105094144 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚351813206026752 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚80338874 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚347637235140180 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788432900666759e+09 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚356879105094144 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚88000940 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚88000940 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 5047248826364 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚389477042213888 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 5315574679552 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚384161587072512 trainer/global_step‚6044 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚347636727186428 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 _step‚492 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚159 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5315455141376 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚88000938 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 114196905 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 114180928 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 119538176 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 202197845 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5065899067392 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 202181866 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚51347134 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5066391897600 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚161 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 _runtime‚4585.371852341 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚356879615927296 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚351813224029696 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 5065899067392 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 114180928 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚342589478360064 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚351813224029696 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚351813206026752 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚51347132 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚389477185692160 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚384161611012608 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 _step‚492 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚159 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5315455141376 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚88000938 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 114196905 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 114180928 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 119538176 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 202197845 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5065899067392 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 202181866 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚51347134 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5066391897600 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚161 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 _runtime‚4585.371852341 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚356879615927296 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚351813224029696 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 5065899067392 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 114180928 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚342589478360064 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚351813224029696 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚351813206026752 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚51347132 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚389477185692160 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚384161611012608 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5047612435772 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 114188230 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 114188230 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚161 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚342624789158912 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 XJDeviceStatsMonitor.on_validation_batcºg6±ðh_end/allocated_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚88011746 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚347672892509076 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚159 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚356916300249600 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 202215953 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚356916300249600 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚384199486746112 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚389515426552320 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884329015529597e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5066757024256 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 114188230 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚161 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚342624789158912 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚88011746 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚347672892509076 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚159 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚356916300249600 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 202215953 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚356916300249600 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚384199486746112 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚389515426552320 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884329015529597e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5066757024256 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5047612435772ít/P¡‚(:’ ‰¤åÔ¸Èó gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚156.396' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.279199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹€ü*¢‚(:Ÿ ‰¤åÔò¯Š‘ disk.rbd0.out‚0$ memory_percent‚94.20797824859619* proc.memory.percent‚17.537927627563477 network.recv‚ 1.686148e+06 disk.nvme0n1.in‚ 440.99609375 disk.sda2.out‚0 disk.sda3.out‚ 180.41796875 disk.rbd3.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 180.41796875& disk.nvme0c0n1.out‚26246.974609375( proc.memory.availableMB‚ 948.96484375 proc.cpu.threads‚17 network.sent‚ 1.3602925e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚2.453125" disk.nvme0c0n1.in‚ 440.99609375 disk.drbd1777.in‚0) disk./.usagePercent‚73.87525189093309! disk.dm-0.out‚26246.974609375 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 cpu‚25.361275664628625 disk.rbd0.in‚0 disk.rbd2.out‚137.4580078125 disk.dm-0.in‚ 440.99609375 disk.sda.out‚ 180.41796875 disk.rbd2.in‚0" proc.memory.rssMB‚ 2873.4140625 disk.drbd1777.out‚0$ disk.nvme0n1.out‚26246.974609375% disk./.usageGB‚1100.7946395874023 disk.rbd1.in‚0 disk.sda2.in‚0ξø2£- _wandbruntime‚4598  _runtime‚4598†q@î–:¤š¦ 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9892010688781738 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1930972933769226 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2636098563671112 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.40344834327697754 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2084498554468155 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 *loss/val_total_loss‚43.326070384992356 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2886606752872467 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_6_of_6‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03988095372915268 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3345274329185486 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 jet/accuracy_5_of_6‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4068410098552704 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24478740990161896 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07957194745540619 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38761937618255615 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.33257579803466797 particle/accuracy_4_of_7‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666269302368 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40497076511383057 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4404761791229248 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9858062863349915 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19542992115020752 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4215543866157532 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45035120844841003 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1959628313779831 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 particle/accuracy_2_of_2‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4487803876399994 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06280788034200668 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07839125394821167 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19631502032279968 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4417952597141266 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23985165357589722 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.45225784182548523 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47504308819770813 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 -jet/accuracy_4_of_4‚0.0051451632753014565 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23162110149860382 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 epoch‚4 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30779939889907837 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.992252767086029 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18693524599075317 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1966695487499237 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0017636684933677316 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3492507040500641 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1756906509399414 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.45830848813056946 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15817569196224213 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9824561476707458 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.24425621330738068 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3611111342906952 +jet/accuracy_3_of_6‚0.09615384787321091 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_4‚0.36771371960639954 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9872470498085022 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.23732468485832214 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 jet/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9101654887199402 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46738550066947937 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4430786669254303 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1756906509399414 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.28390806913375854 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27407407760620117 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 ( Purity/3FRt4SRqqt*FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4193900227546692 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.48765432834625244 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4577201008796692 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4120299518108368 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4635147750377655 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 trainer/global_step‚6044 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04540516436100006 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 (particle/accuracy‚0.8477957844734192 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.014150943607091904 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.42346465587615967 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4874052107334137 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00749084772542119 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 )jet/accuracy_2_of_6‚0.557692289352417 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37150296568870544 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.21743296086788177 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 particle/accuracy_5_of_6‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45035120844841003 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09537176787853241 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.8998202681541443 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02063491940498352 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.36607566475868225 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3775533437728882 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.46470436453819275 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02176935411989689 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9901341199874878 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5680299997329712 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19747532904148102 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2606322169303894 particle/accuracy_1_of_5‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4833254814147949 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4935681223869324 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 +jet/accuracy_4_of_6‚0.03846153989434242 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17943742871284485 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9798850417137146 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.45400601625442505 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5000000596046448 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9734129905700684 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3541666567325592 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqtÉN¹z‹X_purity‚0.5404137969017029 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9905089735984802 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.04605263099074364 *jet/accuracy_1_of_3‚0.8540828824043274 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666269302368 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23582309484481812 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878787517547607 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2946428656578064 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3611111342906952 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 jet/accuracy_7_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4020855128765106 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.33099403977394104 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011664374731481075 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.49908652901649475 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9776113629341125 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2595188319683075 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7"Purity/1FRt0SRqqt1FBt/event_purity‚0.27219158411026 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.34501907229423523 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1879478096961975 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4907407760620117 particle/accuracy_3_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9829072952270508 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3779101073741913 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ( Purity/3FRt4SRqqt0FBt/FRt_purity‚0.5 particle/accuracy_1_of_7‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4439813792705536 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2252545952796936 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 +jet/accuracy_3_of_4‚0.07050298154354095 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1689854860305786 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.45905736088752747 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020792962983250618 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4795284867286682 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9894425868988037 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.48852112889289856 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 jet/accuracy_2_of_7‚0.5 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5064770579338074 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347776651382 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01681693270802498 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6876416802406311 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9384344220161438 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3592592179775238 particle/accuracy_3_of_6‚1 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027095554396510124 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.22736626863479614 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5956959128379822 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 _runtime‚4598.445152673 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9889124035835266 particle/accuracy_1_of_3‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 _step‚494 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3055555522441864 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 (particle/f_score‚0.36667829751968384 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 particle/accuracy_2_of_6‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34108734130859375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4937500059604645 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2597484290599823 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4298245906829834 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,jet/accuracy_4_of_5‚0.015456990338861942 particle/accuracy_2_of_5‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1487666815519333 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22555764019489288 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.27339181303977966 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.45319828391075134 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5152914524078369 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4316754937171936 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962127804756165 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3972868025302887 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5615783929824829 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24082742631435394 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ( Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2399691492319107 jet/accuracy_1_of_6‚1 particle/accuracy_5_of_7‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5447632074356079 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0330999419093132 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4301171600818634 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011342247016727924 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43656468391418457 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8132672309875488 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9834997057914734 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 jet/accuracy_3_of_7‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01889270730316639 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.29434648156166077 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6052631735801697 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2481290102005005 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09700661152601242 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1924004703760147 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4000000059604645 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4234204888343811 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05292468145489693 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37782737612724304 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_2_of_5‚0.4600870609283447 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2275642603635788 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.36549559235572815 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 particle/accuracy_3_of_4‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5219437479972839 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.243660569190979 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3981468677520752 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +particle/specificity‚0.9524185061454773 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.35914841294288635 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19531592726707458 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884329137383583e+09 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857735633850098 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.3360360562801361 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5440605878829956 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9905274510383606 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11486468464136124 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9873135089874268 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +particle/sensitivity‚0.2824418842792511 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 *jet/accuracy_1_of_2‚0.7388740181922913 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6360584497451782 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4012351930141449 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_3‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 particle/accuracy_4_of_6‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14988993108272552 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.41272494196891785 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5584370493888855 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0295890960842371 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3795138895511627 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4444443881511688 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006015368737280369 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3839404881000519 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24003556370735168 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.29729729890823364 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.28956228494644165 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.2012578547000885 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976977705955505 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_2_of_3‚0.23245550692081451 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4881078600883484 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9946199059486389 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25341489911079407 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2361142337322235 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4480287730693817 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3681625723838806 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5934027433395386 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5033084750175476 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5042548775672913 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333335876464844 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9433333873748779 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2368774116039276 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03682170808315277 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3598484694957733 +jet/accuracy_2_of_2‚0.09951949864625931 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.22520175576210022 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.17555557191371918 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897217750549316 +jet/accuracy_3_of_5‚0.10366743803024292 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 jet/accuracy_6_of_7‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19554351270198822 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9883828163146973 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3857654631137848 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02230716310441494 particle/accuracy_4_of_5‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 particle/accuracy_2_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0spOFg'¥Ÿ¦ 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9892010688781738 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1930972933769226 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2636098563671112 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.40344834327697754 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2084498554468155 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 *loss/val_total_loss‚43.326070384992356 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2886606752872467 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_6_of_6‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03988095372915268 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3345274329185486 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 jet/accuracy_5_of_6‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4068410098552704 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24478740990161896 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07957194745540619 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38761937618255615 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.33257579803466797 particle/accuracy_4_of_7‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666269302368 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40497076511383057 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4404761791229248 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9858062863349915 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19542992115020752 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4215543866157532 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45035120844841003 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1959628313779831 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 particle/accuracy_2_of_2‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4487803876399994 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06280788034200668 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07839125394821167 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19631502032279968 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4417952597141266 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23985165357589722 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.45225784182548523 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47504308819770813 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 -jet/accuracy_4_of_4‚0.0051451632753014565 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23162110149860382 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 epoch‚4 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30779939889907837 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.992252767086029 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18693524599075317 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1966695487499237 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0017636684933677316 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3492507040500641 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1756906509399414 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.45830848813056946 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15817569196224213 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9824561476707458 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.24425621330738068 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3611111342906952 +jet/accuracy_3_of_6‚0.09615384787321091 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_4‚0.36771371960639954 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9872470498085022 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.23732468485832214 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 jet/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9101654887199402 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46738550066947937 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4430786669254303 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1756906509399414 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.28390806913375854 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27407407760620117 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 ( Purity/3FRt4SRqqt*FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4193900227546692 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.48765432834625244x‚@°¿k ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4577201008796692 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4120299518108368 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4635147750377655 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 trainer/global_step‚6044 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04540516436100006 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 (particle/accuracy‚0.8477957844734192 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.014150943607091904 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.42346465587615967 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4874052107334137 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00749084772542119 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 )jet/accuracy_2_of_6‚0.557692289352417 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37150296568870544 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.21743296086788177 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 particle/accuracy_5_of_6‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45035120844841003 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09537176787853241 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.8998202681541443 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02063491940498352 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.36607566475868225 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3775533437728882 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.46470436453819275 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02176935411989689 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9901341199874878 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5680299997329712 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19747532904148102 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2606322169303894 particle/accuracy_1_of_5‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4833254814147949 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4935681223869324 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 +jet/accuracy_4_of_6‚0.03846153989434242 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17943742871284485 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9798850417137146 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.45400601625442505 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5000000596046448 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9734129905700684 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3541666567325592 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5404137969017029 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9905089735984802 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.04605263099074364 *jet/accuracy_1_of_3‚0.8540828824043274 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666269302368 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23582309484481812 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878787517547607 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2946428656578064 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3611111342906952 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 jet/accuracy_7_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4020855128765106 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.33099403977394104 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011664374731481075 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.49908652901649475 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9776113629341125 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2595188319683075 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7"Purity/1FRt0SRqqt1FBt/event_purity‚0.27219158411026 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.34501907229423523 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1879478096961975 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4907407760620117 particle/accuracy_3_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9829072952270508 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3779101073741913 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ( Purity/3FRt4SRqqt0FBt/FRt_purity‚0.5 particle/accuracy_1_of_7‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4439813792705536 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2252545952796936 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 +jet/accuracy_3_of_4‚0.07050298154354095 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1689854860305786 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.45905736088752747 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020792962983250618 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4795284867286682 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9894425868988037 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.48852112889289856 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 jet/accuracy_2_of_7‚0.5 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5064770579338074 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347776651382 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01681693270802498 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6876416802406311 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9384344220161438 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3592592179775238 particle/accuracy_3_of_6‚1 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027095554396510124 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.22736626863479614 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5956959128379822 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 _runtime‚4598.445152673 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9889124035835266 particle/accuracy_1_of_3‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 _step‚494 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3055555522441864 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 (particle/f_score‚0.36667829751968384 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 particle/accuracy_2_of_6‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34108734130859375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4937500059604645 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2597484290599823 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4298245906829834 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,jet/accuracy_4_of_5‚0.015456990338861942 particle/accuracy_2_of_5‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1487666815519333 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22555764019489288 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.27339181303977966 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.45319828391075134 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5152914524078369 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4316754937171936 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962127804756165 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3972868025302887 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5615783929824829 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24082742631435394 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ( Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2399691492319107 jet/accuracy_1_of_6‚1 particle/accuracy_5_of_7‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5447632074356079 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0330999419093132 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4301171600818634 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011342247016727924 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43656468391418457 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8132672309875488 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9834997057914734 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 jet/accuracy_3_of_7‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01889270730316639 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.29434648156166077 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6052631735801697 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2481290102005005 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09700661152601242 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1924004703760147 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4000000059604645 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4234204888343811 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05292468145489693 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37782737612724304 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_2_of_5‚0.4600870609283447 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2275642603635788 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.36549559235572815 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 particle/accuracy_3_of_4‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5219437479972839 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.243660569190979 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3981468677520752 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +particle/specificity‚0.9524185061454773 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.35914841294288635 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19531592726707458 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884329137383583e+09 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857735633850098 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.3360360562801361 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5440605878829956 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9905274510383606 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11486468464136124 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9873135089874268 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +particle/sensitivity‚0.2824418842792511 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 *jet/accuracy_1_of_2‚0.7388740181922913 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6360584497451782 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4012351930141449 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_3‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 particle/accuracy_4_of_6‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14988993108272552 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.41272494196891785 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5584370493888855 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0295890960842371 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3795138895511627 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4444443881511688 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006015368737280369 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3839404881000519 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24003556370735168 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.29729729890823364 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.28956228494644165 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.2012578547000885 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976977705955505 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_2_of_3‚0.23245550692081451 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4881078600883484 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9946199059486389 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25341489911079407 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2361142337322235 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4480287730693817 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3681625723838806 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5934027433395386 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5033084750175476 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5042548775672913 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333335876464844 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9433333873748779 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2368774116039276 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03682170808315277 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3598484694957733 +jet/accuracy_2_of_2‚0.09951949864625931 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.22520175576210022 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.17555557191371918 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897217750549316 +jet/accuracy_3_of_5‚0.10366743803024292 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 jet/accuracy_6_of_7‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19554351270198822 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9883828163146973 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3857654631137848 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02230716310441494 particle/accuracy_4_of_5‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 particle/accuracy_2_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0îoŒ“¦‚(:ˆ ˜¤åÔ¸Ùß gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚149.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾vL&§‚(:› ˜¤åÔïÌ€( disk.nvme0c0n1.out‚26819.49365234375 disk.rbd2.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚26819.49365234375* proc.memory.percent‚17.537927627563477 disk.drbd1777.out‚0& disk.nvme0n1.out‚26819.49365234375 disk.rbd1.out‚2.453125" disk.nvme0c0n1.in‚ 440.99609375 disk.sda1.out‚0 cpu‚25.369132972494906 network.recv‚ 1.689881e+06 disk.dm-1.out‚ 180.953125 disk.sda2.in‚0 disk.rbd2.out‚137.4580078125 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.sda.out‚ 180.953125 proc.cpu.threads‚17 disk.dm-0.in‚ 440.99609375& proc.memory.availableMB‚ 1424.40625" proc.memory.rssMB‚ 2873.4140625 disk.sda3.out‚ 180.953125 disk.sda.in‚0$ memory_percent‚91.30611419677734 disk.rbd3.out‚0 network.sent‚ 1.3708722e+07 disk.dm-1.in‚0 disk.nvme0n1.in‚ 440.99609375 disk.sda2.out‚0 disk.rbd0.in‚0( disk./.usagePercent‚73.8753768226098$ disk./.usageGB‚1100.796501159668 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 üË‹2¨- _wandbruntime‚4605  _runtime‚4605‹Ç_FÉ©Ã ' lr-AdamW/pg2‚0.00013285883472219088 trainer/global_step‚6049 % _timestamp‚1.7884329209194531e+09 _runtime‚4605.624191255 _step‚495 ' lr-AdamW/pg1‚0.00013285883472219088ò5\ÎªÈ ' lr-AdamW/pg1‚0.00013285883472219088 ' lr-AdamW/pg2‚0.00013285883472219088 trainer/global_step‚6049 % _timestamp‚1.7884329209194531e+09 _runtime‚4605.624191255 _step‚495ïx‹RÁ2«- _wandbruntime‚4605  _runtime‚4605¶¹¿ÔU ¬ÚO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357629044389888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5074131439104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114376559 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29074003 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390262754345984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5324280113664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348359325469378 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114356521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357628370024448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5074787801600 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202564807 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114356521 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80544450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.788432920920157e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88208286 TGDeviceStatsMoni[D‹tor.on_train_batch_start/inactive_split.large_pool.freed‚51470860 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343303224554496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88208288 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357629044389888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114376559 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5055446477126 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88208288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚357628370024448 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352554238585344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88208286 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚413 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202244096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202584847 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384938474232320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80544865 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚415 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51470862 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5056100914882 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352554256588288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430>éOå'­ßO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5074131439104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114376559 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29074003 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390262754345984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5324280113664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348359325469378 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114356521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357628370024448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5074787801600 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202564807 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114356521 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80544450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.788432920920157e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88208286 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51470860 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343303224554496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88208288 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357629044389888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114376559 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5055446477126 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88208288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚357628370024448 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352554238585344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88208286 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚413 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202244096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202584847 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384938474232320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80544865 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚415 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51470862 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5056100914882 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352554256588288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357629044389888ð”Y(2®- _wandbruntime‚4606  _runtime‚4606 ƒÆœë&¯åM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80557548 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚385000124969984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348414374557786 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5056917350054 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88222302 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357686216869888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390325251739648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114374975 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390325047411712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88222304 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114395039 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6049 _runtime‚4606.507571951 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202617343 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352610609627136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 204327936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51479030 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5325126769664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80557114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180387840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚497 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114395039 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚343358128685056 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5074950866944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88222302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348415046035110 % _timestamp‚1.7884329218024096e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352610609627136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5075607242752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚434 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385000101029888 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202617343 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357686216869888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29078086 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5324946381824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5056262912090 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5074950866944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352610591624192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5075607242752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202597277 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51479028 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29078518 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352610591624192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88222304V†~P°êM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352610591624192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88222304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80557548 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚385000124969984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348414374557786 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5056917350054 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88222302 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357686216869888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390325251739648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114374975 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390325047411712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88222304 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114395039 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6049 _runtime‚4606.507571951 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202617343 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352610609627136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 204327936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51479030 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5325126769664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80557114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180387840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.largeÔ¸Uâ _pool.freed‚0 _step‚497 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114395039 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚343358128685056 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5074950866944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88222302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348415046035110 % _timestamp‚1.7884329218024096e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352610609627136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5075607242752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚434 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385000101029888 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202617343 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357686216869888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29078086 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5324946381824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5056262912090 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5074950866944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352610591624192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5075607242752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202597277 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51479028 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29078518 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680ñ ~k 2±- _wandbruntime‚4606  _runtime‚4606PÙMëY²Ó 0loss/FBt1/assignment_loss‚0.1422911818172246 % _timestamp‚1.7884329218040726e+09 _runtime‚4606.508710475 _step‚498 1loss/FBt2/assignment_loss‚0.06733968739198255 .loss/FRt3/detection_loss‚2.891319626569748 /loss/FRt4/detection_loss‚2.4143767642213945 /loss/FBt3/detection_loss‚3.3411080752100264 trainer/global_step‚6049 0loss/FRt3/assignment_loss‚3.4409000635147096 1loss/FBt3/assignment_loss‚0.02678997488692403 .loss/FRt2/detection_loss‚2.656306551817136 0loss/SRqqt2/detection_loss‚2.274862345866859 0loss/SRqqt3/detection_loss‚2.354152754374913 /loss/FRt2/assignment_loss‚2.957958996295929 2loss/SRqqt1/assignment_loss‚1.0439131314201013 2loss/SRqqt3/assignment_loss‚1.6054839896304267 0loss/SRqqt4/detection_loss‚1.929254749417305 2loss/SRqqt2/assignment_loss‚0.9855802361853421 .loss/FRt1/detection_loss‚1.565173067439061 %loss/total_loss‚42.73844991976004 /loss/FRt4/assignment_loss‚3.610130291035835 1loss/FBt4/assignment_loss‚0.28109892333547276 epoch‚5 /loss/FBt4/detection_loss‚2.3725558718045554 2loss/SRqqt4/assignment_loss‚1.2258571872115136 1loss/SRqqt1/detection_loss‚1.0670925010527883 /loss/FBt1/detection_loss‚0.6955327276581761 0loss/FRt1/assignment_loss‚1.0113162008615642 .loss/FBt2/detection_loss‚2.778055020741054Ÿ‹8 ^³Ø 0loss/FRt3/assignment_loss‚3.4409000635147096 1loss/FBt3/assignment_loss‚0.02678997488692403 .loss/FRt2/detection_loss‚2.656306551817136 0loss/SRqqt2/detection_loss‚2.274862345866859 0loss/SRqqt3/detection_loss‚2.354152754374913 /loss/FRt2/assignment_loss‚2.957958996295929 2loss/SRqqt1/assignment_loss‚1.0439131314201013 2loss/SRqqt3/assignment_loss‚1.6054839896304267 0loss/SRqqt4/detection_loss‚1.929254749417305 2loss/SRqqt2/assignment_loss‚0.9855802361853421 .loss/FRt1/detection_loss‚1.565173067439061 %loss/total_loss‚42.73844991976004 /loss/FRt4/assignment_loss‚3.610130291035835 1loss/FBt4/assignment_loss‚0.28109892333547276 epoch‚5 /loss/FBt4/detection_loss‚2.3725558718045554 2loss/SRqqt4/assignment_loss‚1.2258571872115136 1loss/SRqqt1/detection_loss‚1.0670925010527883 /loss/FBt1/detection_loss‚0.6955327276581761 0loss/FRt1/assignment_loss‚1.0113162008615642 .loss/FBt2/detection_loss‚2.778055020741054 0loss/FBt1/assignment_loss‚0.1422911818172246 % _timestamp‚1.7884329218040726e+09 _runtime‚4606.508710475 _step‚498 1loss/FBt2/assignment_loss‚0.06733968739198255 .loss/FRt3/detection_loss‚2.891319626569748 /loss/FRt4/detection_loss‚2.4143767642213945 /loss/FBt3/detection_loss‚3.3411080752100264 trainer/global_step‚6049ò抸Ǔ´‚(:ˆ §¤åÔž•ï gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞCL°$µ‚(:™ §¤åÔÛâ«‘& disk.nvme0n1.out‚26918.59521484375 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 440.99609375 disk.sda2.out‚0 network.recv‚ 1.69548e+06 disk.sda1.out‚0" proc.memory.rssMB‚ 2873.4140625 disk.sda.in‚0 disk.sda3.out‚181.5625 disk.rbd3.in‚0 disk.dm-0.in‚ 440.99609375 disk.rbd0.in‚0$ memory_percent‚91.30561351776123 disk.rbd3.out‚0 disk.rbd1.out‚2.453125 network.sent‚ 1.3832551e+07) disk./.usagePercent‚73.87538040671528% disk./.usageGB‚1100.7965545654297# disk.dm-0.out‚26918.59521484375 disk.sda1.in‚0 disk.nvme0n1.in‚ 440.99609375 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚26918.59521484375 disk.rbd0.out‚0* proc.memory.percent‚17.537927627563477 disk.dm-1.in‚0 disk.dm-1.out‚181.5625 disk.sda2.in‚0! disk.rbd2.out‚137.56005859375) proc.memory.availableMB‚ 1424.48828125 cpu‚25.36808750176681 disk.sda.out‚181.5625 proc.cpu.threads‚17Ô7<쓶‚(:ˆ ¶¤åÔƪï gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.087' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.4174 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆÚ#·‚(:˜ ¶¤åÔ¤‰î% disk./.usageGB‚1100.7945137023926 disk.rbd1.out‚2.453125" disk.nvme0c0n1.in‚ 440.99609375 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 1424.16796875 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 440.99609375 disk.sda.in‚0" proc.memory.rssMB‚ 2873.4140625 network.sent‚ 1.3939021e+07 network.recv‚ 1.699544e+06) disk./.usagePercent‚73.87524344268445 disk.rbd3.in‚0 proc.cpu.threads‚17# disk.nvme0n1.out‚27490.94921875 disk.rbd2.in‚0 disk.sda3.in‚0 disk.dm-0.out‚27490.94921875% disk.nvme0c0n1.out‚27490.94921875 disk.sda.out‚ 181.890625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.rbd2.out‚137.56005859375 disk.sda3.out‚ 181.890625$ memory_percent‚91.30756855010986 disk.dm-1.out‚ 181.890625 disk.sda2.in‚0 disk.drbd1777.out‚0 cpu‚25.367026911470887 disk.sda1.out‚0 disk.nvme0n1.in‚ 440.99609375* proc.memory.percent‚17.537927627563477^Å>2¸- _wandbruntime‚4640  _runtime‚4640xåÌǹÁ _step‚499 & lr-AdamW/pg1‚0.0001312455988349091 & lr-AdamW/pg2‚0.0001312455988349091 trainer/global_step‚6099 % _timestamp‚1.7884329560156798e+09 _runtime‚4640.720451624dR}ÌºÆ trainer/global_step‚6099 % _timestamp‚1.7884329560156798e+09 _runtime‚4640.720451624 _step‚499 & lr-AdamW/pg1‚0.0001312455988349091 & lr-AdamW/pg2‚0.0001312455988349091ó*²˜t2»- _wandbruntime‚4640  _runtime‚4640Zúô‘á'¼ÛO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360487682138112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81177935 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115282573 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4640.721331521 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204211699 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚6099 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115302611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355371890527744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51879260 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388021011115520 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5366821201408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351144695859670 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5096936254290 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5366993213952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346048414043136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204191659 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29298675 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360487682138112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393388004329472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355371908530688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351145367336786 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355371908530688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388020987175424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5115773607424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5115773607424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115302611 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29299116 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5115117244928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360487007772672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81178378 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360487007772672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5115117244928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346048431082496 % _timestamp‚1.7884329560164416e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88909086 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038ÀI%Læ'½àO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360487682138112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81177935 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115282573 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4640.721331521 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204211699 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚6099 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115302611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355371890527744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51879260 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388021011115520 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5366821201408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351144695859670 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5096936254290 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5366993213952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346048414043136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204191659 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29298675 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360487682138112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393388004329472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355371908530688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351145367336786 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355371908530688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388020987175424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5115773607424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5115773607424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115302611 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29299116 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5115117244928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360487007772672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81178378 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360487007772672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5115117244928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346048431082496 % _timestamp‚1.7884329560164416e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88909086 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2ôf S¹2¾- _wandbruntime‚4641  _runtime‚4641¦<åÞª¿ãM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115301027 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29303160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5367667586048 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88923104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 MGDeviceStatsMÿN?!onitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360544854618112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204224129 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88923102 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚4641.383953145 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81190588 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29303611 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351200416425194 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204244195 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _step‚501 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5115936672768 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204244195 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88923104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355428261569536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚6099 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393450305499136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5116593048576 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5115936672768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81191041 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 # _timestamp‚1.78843295667877e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115321091 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351201087902518 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚355428243566592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355428261569536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5116593048576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430[£“î&ÀèM PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88923104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355428261569536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚6099 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393450305499136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5116593048576 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5115936672768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81191041 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 # _timestamp‚1.78843295667877e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115321091 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351201087902518 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚355428243566592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355428261569536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5116593048576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115301027 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29303160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5367667586048 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88923104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360544854618112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204224129 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88923102 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚4641.383953145 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81190588 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29303611 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351200416425194 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204244195 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _step‚501 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5115936672768 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204244195 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚391103547998720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5156759413248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89609886 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89609888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5409823500800 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚348793637610496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚363345645520896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116208625 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944‘Qöæ'ÍàO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4675.803148056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚353930737727078 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116208625 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358189542470144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5409655682560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚363345645520896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116208625 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚504 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚363345645520896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205838551 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884329910981622e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89609888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29523977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚391103547998720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5156759413248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89609886 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89609888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5409823500800 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚348793637610496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚348848524701696 trainer/global_step‚6149 $ _timestamp‚1.788432991759528e+09 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5410680895488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116227079 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚505 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5410517284864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358245913511936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚396575879631872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52295828 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205850981 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚363403492366336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29528468 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚391165198736384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205850981 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52295830 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5156922478592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚363403492366336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358245895508992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81824735 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205871047 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89623902 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116247143 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5156922478592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚353986458292602 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358245913511936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353987129769926 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5138588028870 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89623904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5137933590906 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116247143 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚391165174796288 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89623902 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29528905 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚363402817987584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205871047 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81824296FBöï&ÐéM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚363403492366336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358245895508992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81824735 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205871047 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89623902 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116247143 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5156922478592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚353986458292602 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358245913511936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353987129769926 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5138588028870 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89623904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5137933590906 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116247143 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚391165174796288 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89623902 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29528905 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚363402817987584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205871047 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81824296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _runtime‚4676.464802873 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚348848524701696 trainer/global_step‚6149 $ _timestamp‚1.788432991759528e+09 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5410680895488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116227079 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚505 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5410517284864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358245913511936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚396575879631872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52295828 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205850981 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚363403492366336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29528468 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚391165198736384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205850981 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52295830 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5156922478592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ù«¯È2Ñ- _wandbruntime‚4676  _runtime‚4676½ÍüOÎÒÕ 1loss/FBt1/assignment_loss‚0.07377058924165321 1loss/SRqqt2/detection_loss‚1.9870983286036386 epoch‚5 trainer/global_step‚6149 /loss/FBt1/detection_loss‚0.5726367261381365 /lo%ÊrÄss/FRt1/detection_loss‚1.8293564902957191 /loss/FRt2/detection_loss‚2.3871039192784917 0loss/FRt4/assignment_loss‚3.4008439250712126 2loss/FBt3/assignment_loss‚0.007516205651752118 1loss/SRqqt1/detection_loss‚0.9477594923731443 _step‚506 0loss/FRt2/assignment_loss‚2.9649015218019485 .loss/FRt4/detection_loss‚2.212777171494826 $loss/total_loss‚42.2350923985677 % _timestamp‚1.7884329917611246e+09 2loss/SRqqt1/assignment_loss‚1.0236442538632735 .loss/FRt3/detection_loss‚3.566697490842719 1loss/SRqqt3/assignment_loss‚1.853640049357306 .loss/FBt4/detection_loss‚2.367229779561361 .loss/FBt2/detection_loss‚2.567198063646044 0loss/FRt1/assignment_loss‚1.7444851899688893 /loss/FRt3/assignment_loss‚3.108921107492949 2loss/SRqqt4/assignment_loss‚1.4504182372810477 _runtime‚ 4676.46576793 1loss/SRqqt4/detection_loss‚1.6667029832379292 1loss/FBt2/assignment_loss‚0.02190866008692995 2loss/FBt4/assignment_loss‚0.005727313885775705 0loss/SRqqt3/detection_loss‚2.193197463046421 /loss/FBt3/detection_loss‚2.7780727318355014 1loss/SRqqt2/assignment_loss‚1.503484704511033z€¸`ÓÚ 0loss/FRt4/assignment_loss‚3.4008439250712126 2loss/FBt3/assignment_loss‚0.007516205651752118 1loss/SRqqt1/detection_loss‚0.9477594923731443 _step‚506 0loss/FRt2/assignment_loss‚2.9649015218019485 .loss/FRt4/detection_loss‚2.212777171494826 $loss/total_loss‚42.2350923985677 % _timestamp‚1.7884329917611246e+09 2loss/SRqqt1/assignment_loss‚1.0236442538632735 .loss/FRt3/detection_loss‚3.566697490842719 1loss/SRqqt3/assignment_loss‚1.853640049357306 .loss/FBt4/detection_loss‚2.367229779561361 .loss/FBt2/detection_loss‚2.567198063646044 0loss/FRt1/assignment_loss‚1.7444851899688893 /loss/FRt3/assignment_loss‚3.108921107492949 2loss/SRqqt4/assignment_loss‚1.4504182372810477 _runtime‚ 4676.46576793 1loss/SRqqt4/detection_loss‚1.6667029832379292 1loss/FBt2/assignment_loss‚0.02190866008692995 2loss/FBt4/assignment_loss‚0.005727313885775705 0loss/SRqqt3/detection_loss‚2.193197463046421 /loss/FBt3/detection_loss‚2.7780727318355014 1loss/SRqqt2/assignment_loss‚1.503484704511033 1loss/FBt1/assignment_loss‚0.07377058924165321 1loss/SRqqt2/detection_loss‚1.9870983286036386 epoch‚5 trainer/global_step‚6149 /loss/FBt1/detection_loss‚0.5726367261381365 /loss/FRt1/detection_loss‚1.8293564902957191 /loss/FRt2/detection_loss‚2.3871039192784917úZ†<§Ô‚(:’ ã¤åÔƒãæ gpu.0.gpu‚59.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.44' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.887999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜{Þl,Õ‚(:¡ ã¤åÔïØ‘ disk.sda2.in‚0! disk.rbd2.out‚139.22802734375$ memory_percent‚91.32168292999268" proc.memory.rssMB‚ 2873.4140625 network.recv‚ 1.712047e+06% disk./.usageGB‚1100.7954902648926! disk.dm-0.out‚27689.767578125$ disk.nvme0n1.out‚27689.767578125 disk.sda.in‚0 disk.rbd1.out‚2.453125* proc.memory.percent‚17.537927627563477 disk.drbd1777.out‚0" disk.nvme0c0n1.in‚ 440.99609375 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda3.out‚ 183.73046875) proc.memory.availableMB‚ 1421.85546875 cpu‚25.363981676981762& disk.nvme0c0n1.out‚27689.767578125 network.sent‚ 1.4056746e+07 disk.rbd3.in‚0 disk.nvme0n1.in‚ 440.99609375 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 183.73046875 disk.rbd0.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.87530898061323 disk.dm-1.out‚ 183.73046875 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 440.99609375IÂAÓ“Ö‚(:ˆ ò¤åÔÁóÜ gpu.0.gpu‚45.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þj¹ÿ2ׂ(:§ ò¤åÔÅÑá" disk.nvme0c0n1.in‚ 451.22265625 disk.rbd0.in‚0! disk.rbd2.out‚139.24365234375 disk.nvme0n1.in‚ 451.22265625 disk.rbd1.in‚0" proc.memory.rssMB‚ 2873.4140625 disk.sda3.in‚0 disk.sda3.out‚ 184.08984375 disk.rbd2.in‚0 disk.drbd1777.out‚0# disk.dm-0.out‚27789.20068359375 disk.sda.out‚ 184.08984375 disk.dm-1.out‚ 184.08984375& disk.nvme0n1.out‚27789.20068359375 disk.rbd1.out‚2.453125% disk./.usageGB‚1100.7956275939941 disk.rbd3.in‚0 disk.sda1.out‚0 cpu‚25.362998858634494( disk.nvme0c0n1.out‚27789.20068359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 451.22265625$ memory_percent‚91.32192134857178) proc.memory.availableMB‚ 1421.81640625 proc.cpu.threads‚17) disk./.usagePercent‚73.87531819688445 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.537927627563477 network.sent‚ 1.4164119e+07 network.recv‚ 1.717303e+06„­‚žØ‚(:“ ¥åÔ³‹ˆŽ gpu.0.gpu‚46.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.908' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.581599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fqý''Ù‚(:œ ¥åԥР) proc.memory.availableMB‚ 1421.55078125 proc.cpu.threads‚17 network.sent‚ 1.4169319e+07% disk./.usageGB‚1100.7957305908203 disk.rbd0.out‚0 disk.rbd2.out‚ 139.392578125 disk.sda3.out‚ 184.953125 disk.rbd1.in‚0 disk.sda2.in‚0 cpu‚25.362001988225384$ memory_percent‚91.32354259490967 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 574.5546875! disk.nvme0c0n1.in‚ 574.5546875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.537927627563477 disk.rbd3.in‚0 disk.dm-0.in‚ 574.5546875 network.recv‚ 1.720343e+06) disk./.usagePercent‚73.87532510908788# disk.dm-0.out‚27888.30615234375 disk.sda1.in‚0( disk.nvme0c0n1.out‚27888.30615234375" proc.memory.rssMB‚ 2873.4140625 disk.rbd0.in‚0 disk.sda.out‚ 184.953125 disk.sda1.out‚0& disk.nvme0n1.out‚27888.30615234375 disk.sda3.in‚0 disk.dm-1.out‚ 184.953125 disk.rbd1.out‚2.453125 disk.sda2.out‚0"%ø2Ú- _wandbruntime‚4710  _runtime‚4710CY™ÆÛÀ _step‚507 & lr-AdamW/pg1‚0.0001280082516567962 & lr-AdamW/pg2‚0.0001280082516567962 trainer/global_step‚6199 $ _timestamp‚1.788433025736113e+09 _runtime‚4710.440860605G_ËÜÅ & lr-AdamW/pg1‚0.0001280082516567962 & lr-AdamW/pg2‚0.0001280082516567962 trainer/global_step‚6199 $ _timestamp‚1.788433025736113e+09 _runtime‚4710.440860605 _step‚507ûÙävâ2Ý- _wandbruntime‚4710  _runtime‚4710ÿ5;á'ÞÛO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5178606933106 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207445363 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366204283269120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5197088856576 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29749221 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚6199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366204283269120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399638702015488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5452617133568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚351538827099136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884330257368617e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366204957634560 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚361007212415488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90310686 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117134677 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90310688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5452455606784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361007194412544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356717451071602 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚351538844138496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399638516548608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117154715 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207465403 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117154715 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90310688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ÕÞ¸æ'ßàO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82445715 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5197088856576 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90310686 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117134677 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90310688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5452455606784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361007194412544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356717451071602 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚351538844138496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399638516548608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117154715 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207465403 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117154715 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90310688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5178606933106 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207445363 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366204283269120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5197088856576 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29749221 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚6199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366204283269120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399638702015488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5452617133568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚351538827099136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884330257368617e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366204957634560 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚361007212415488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207497899 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52704230 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356772500160010 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29754128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117153131 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚6199 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82457930 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5197908284416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366262130114560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366262130114560 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366261455735808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117153131 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399701216534528 _runtime‚ 4711.10106471 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90324702 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5198564660224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198564660224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5453319401472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399701031080960 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _step‚509 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90324704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207477833 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207477833 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90324704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5197908284416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29753702 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207497899 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117173195 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356773171637334 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5179423368278W„mãï&âéM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207497899 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117173195 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356773171637334 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5179423368278 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚361063547451392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117173195 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚351593731229696 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207497899 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52704230 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356772500160010 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29754128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117153131 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚6199 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82457930 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5197908284416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366262130114560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366262130114560 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366261455735808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117153131 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399701216534528 _runtime‚ 4711.10106471 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90324702 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5198564660224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198564660224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5453319401472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399701031080960 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _step‚509 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90324704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207477833 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207477833 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90324704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5197908284416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29753702 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ý9r¶2ã- _wandbruntime‚4711  _runtime‚4711 Oñ4^äØ 2loss/SRqqt3/assignment_loss‚1.8144586614798754 1loss/FBt1/assignment_loss‚0.13388756519424527 /loss/FRt2/detection_loss‚2.4788459383923076 1loss/SRqqt3/detection_loss‚2.2528917744755743 /loss/FRt1/assignment_loss‚1.174697152355261 2loss/SRqqt1/assignment_loss‚0.9443447919872899 2loss/SRqqt4/assignment_loss‚1.5952726912498474 /loss/FRt3/detection_loss‚2.7407869862185583 _runtime‚4711.102051999 0loss/FBt2/assignment_loss‚0.1612345423547797 2loss/FBt3/assignment_loss‚0.054985626018606126 /loss/FRt4/detection_loss‚2.2066889823751246 /loss/FRt3/assignment_loss‚3.241328118575944 1loss/SRqqt4/detection_loss‚1.5938998940587044 .loss/FBt3/detection_loss‚3.251523514588674 trainer/global_step‚6199 /loss/FRt4/assignment_loss‚2.979454109009276 /loss/FRt1/detection_loss‚1.5793753539522488 /loss/FBt4/detection_loss‚2.6555809463773454 % _timestamp‚1.7884330263974361e+09 1loss/SRqqt2/detection_loss‚1.9232194157356912 /loss/FBt2/detection_loss‚2.3278992440965443 0loss/FRt2/assignment_loss‚2.6317824630633644 0loss/FBt4/assignment_loss‚0.2004782037609922 1loss/SRqqt1/detection_loss‚1.0461212029680609 %loss/total_loss‚41.22785239944931 epoch‚5 _step‚510 2loss/SRqqt2/assignment_loss‚1.4994478308932577 /loss/FBt1/detection_loss‚0.7396473902677372íwÑcåÝ 2loss/SRqqt3/assignment_loss‚1.8144586614798754 1loss/FBt1/assignment_loss‚0.13388756519424527 /loss/FRt2/detection_loss‚2.4788459383923076 1loss/SRqqt3/detection_loss‚2.2528917744755743 /loss/FRt1/assignment_loss‚1.174697152355261 2loss/SRqqt1/assignment_loss‚0.9443447919872899 2loss/SRqqt4/assignment_loss‚1.5952726912498474 /loss/FRt3/detection_loss‚2.7407869862185583 _runtime‚4711.102051999 0loss/FBt2/assignment_loss‚0.1612345423547797 2loss/FBt3/assignment_loss‚0.054985626018606126 /loss/FRt4/detection_loss‚2.2066889823751246 /loss/FRt3/assignment_loss‚3.241328118575944 1loss/SRqqt4/detection_loss‚1.5938998940587044 .loss/FBt3/detection_loss‚3.251523514588674 trainer/global_step‚6199 /loss/FRt4/assignment_loss‚2.979454109009276 /loss/FRt1/detection_loss‚1.5793753539522488 /loss/FBt4/detection_loss‚2.6555809463773454 % _timestamp‚1.7884330263974361e+09 1loss/SRqqt2/detection_loss‚1.9232194157356912 /loss/FBt2/detection_loss‚2.3278992440965443 0loss/FRt2/assignment_loss‚2.6317824630633644 0loss/FBt4/assignment_loss‚0.2004782037609922 1loss/SRqqt1/detection_loss‚1.0461212029680609 %loss/total_loss‚41.22785239944931 epoch‚5 _step‚510 2loss/SRqqt2/assignment_loss‚1.4994478308932577 /loss/FBt1/detection_loss‚0.7396473902677372þµWcæ‚(:’ ¥åÔ¦ø gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.98' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.995999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yl!ç‚(:– ¥åÔØâ­! disk.dm-0.out‚27888.716796875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 574.5546875 network.sent‚ 1.417204e+07 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2873.4140625* proc.memory.percent‚17.537927627563477 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.in‚0 network.recv‚ 1.722057e+06 disk.drbd1777.in‚0 disk.sda.out‚ 185.2109375 disk.sda2.in‚0$ memory_percent‚91.32568836212158 cpu‚25.36101579208035 disk.dm-1.out‚ 185.2109375 disk.rbd1.out‚2.453125 disk.dm-0.in‚ 574.5546875 disk.rbd2.out‚ 139.404296875 disk.sda3.in‚0 disk.sda3.out‚ 185.2109375) proc.memory.availableMB‚ 1421.19921875$ disk./.usageGB‚1100.795753479004$ disk.nvme0n1.out‚27888.716796875 disk.rbd1.in‚0& disk.nvme0c0n1.out‚27888.716796875 proc.cpu.threads‚17) disk./.usagePercent‚73.87532664513309! disk.nvme0c0n1.in‚ 574.5546875 disk.rbd0.in‚07gÊžè‚(:“ Ÿ¥åÔ£ÄË gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.405' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.480999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ퟲH$é‚(:™ Ÿ¥åÔÔ«ý disk.rbd2.out‚ 140.978515625 disk.sda3.in‚0 disk.rbd3.out‚0! disk.nvme0c0n1.in‚ 574.5546875 disk.rbd2.in‚0! disk.dm-0.out‚27987.830078125 disk.sda1.in‚0 disk.dm-1.out‚ 185.69140625 disk.rbd1.in‚0& disk.nvme0c0n1.out‚27987.830078125 disk.sda2.in‚0$ memory_percent‚91.32070541381836 disk.rbd3.in‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2872.4296875 cpu‚25.359977741650077 proc.cpu.threads‚17 network.sent‚ 1.4281085e+07 disk.drbd1777.in‚0 disk.dm-0.in‚ 574.5546875 disk.dm-1.in‚0$ disk.nvme0n1.out‚27987.830078125( disk./.usagePercent‚73.8753245970728 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 574.5546875 disk.sda3.out‚ 185.69140625 disk.sda.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.531919479370117% disk./.usageGB‚1100.7957229614258 disk.sda.out‚ 185.69140625 disk.rbd1.out‚2.453125 disk.sda2.out‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 1422.015625 network.recv‚ 1.728659e+06!2ê- _wandbruntime‚4745  _runtime‚4745Ï`Éëà trainer/global_step‚6249 % _timestamp‚1.7884330606336055e+09 _runtime‚4745.338341035 _step‚511 ' lr-AdamW/pg1‚0.00012638481502426477 ' lr-AdamW/pg2‚0.00012638481502426477bì@ÎìÈ ' lr-AdamW/pg1‚0.00012638481502426477 ' lr-AdamW/pg2‚0.00012638481502426477 trainer/global_step‚6249 % _timestamp‚1.7884330606336055e+09 _runtime‚4745.338341035 _step‚511ÿïÑs;2í- _wandbruntime‚4745  _runtime‚4745³žªcá'îÛO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5219442272514 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209092255 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚369062921017344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91011488 % _timestamp‚1.7884330606343162e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118080767 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53104460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29974422 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363824846354944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91011486 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚359503492939010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5238731024896 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5238731024896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83079324 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354284050666496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚369062921017344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53104462 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91011486 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363824864357888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5238074662400 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚369063595382784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784©oÝÑïàO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚369062921017344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91011488 % _timestamp‚1.7884330606343162e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118080767 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53104460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29974422 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363824846354944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91011486 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚359503492939010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5238731024896 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5238731024896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83079324 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354284050666496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚369062921017344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53104462 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91011486 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363824864357888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5238074662400 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚369063595382784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5219442272514 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209092255 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844€`å ë2ð- _wandbruntime‚4746  _runtime‚4746i3 Õë&ñåM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5220258707686 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118079183 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363881199393792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53112628 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402826598851584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5496350288896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359559213504742 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5238894090240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚369120093484032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118079183 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5239550466048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4746.000958346 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚369120767862784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354338937757696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209104685 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363881217396736 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118099247 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83091971 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83091529 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363881217396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91025502 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29978901 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354338954797056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53112630 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚513 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5238894090240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5496520190976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209124751 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359558542027418 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209124751 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363881199393792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53112628 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402826598851584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5496350288896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359559213504742 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5238894090240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚369120093484032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118079183 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5239550466048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4746.000958346 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚369120767862784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354338937757696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209104685 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363881217396736 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118099247 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83091971 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83091529 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363881217396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91025502 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29978901 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354338954797056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53112630 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚513 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5238894090240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5496520190976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209124751 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414’¹¡2ó- _wandbruntime‚4746  _runtime‚4746¯å…\ôÖ 0loss/FRt1/assignment_loss‚1.4874845640017436 /loss/FRt4/detection_loss‚2.5157776667791256 trainer/global_step‚6249 _step‚514 0loss/FRt2/assignment_loss‚2.8417926090104237 2loss/SRqqt1/assignment_loss‚1.2284498925000662 1loss/FBt2/assignment_loss‚0.12544251880399315 1loss/FBt4/assignment_loss‚0.45433982624672353 0loss/SRqqt3/detection_loss‚2.571147552558354 /loss/FRt1/detection_loss‚1.6857985043181822 .loss/FRt2/detection_loss‚2.509064549491519 0loss/SRqqt4/detection_loss‚1.750491916601147 _runtime‚4746.001935275 .loss/FRt3/detection_loss‚3.010215436590129 .loss/FBt2/detection_loss‚2.354764051735401 .loss/FBt4/detection_loss‚2.759894073009491 1loss/SRqqt3/assignment_loss‚1.942339549852269 2loss/FBt3/assignment_loss‚0.009900501556694507 1loss/SRqqt2/detection_loss‚1.8851473754644394 /loss/FBt1/detection_loss‚0.6819413662577669 /loss/FBt3/detection_loss‚3.0154775977134705 % _timestamp‚1.7884330612973053e+09 0loss/FRt4/assignment_loss‚2.8015918644035565 2loss/SRqqt4/assignment_loss‚1.2604631325230002 epoch‚5 2loss/SRqqt2/assignment_loss‚1.4679121669381856 0loss/FRt3/assignment_loss‚3.7792700323565254 1loss/FBt1/assignment_loss‚0.19498583738252617 1loss/SRqqt1/detection_loss‚0.8972133440547623 $loss/total_loss‚43.2309059301495ŸRåõaõÛ 1loss/SRqqt3/assignment_loss‚1.942339549852269 2loss/FBt3/assignment_loss‚0.009900501556694507 1loss/SRqqt2/detection_loss‚1.8851473754644394 /loss/FBt1/detection_loss‚0.6819413662577669 /loss/FBt3/detection_loss‚3.0154775977134705 % _timestamp‚1.7884330612973053e+09 0loss/FRt4/assignment_loss‚2.8015918644035565 2loss/SRqqt4/assignment_loss‚1.2604631325230002 epoch‚5 2loss/SRqqt2/assignment_loss‚1.4679121669381856 0loss/FRt3/assignment_loss‚3.7792700323565254 1loss/FBt1/assignment_loss‚0.19498583738252617 1loss/SRqqt1/detection_loss‚0.8972133440547623 $loss/total_loss‚43.2309059301495 0loss/FRt1/assignment_loss‚1.4874845640017436 /loss/FRt4/detection_loss‚2.5157776667791256 trainer/global_step‚6249 _step‚514 0loss/FRt2/assignment_loss‚2.8417926090104237 2loss/SRqqt1/assignment_loss‚1.2284498925000662 1loss/FBt2/assignment_loss‚0.12544251880399315 1loss/FBt4/assignment_loss‚0.45433982624672353 0loss/SRqqt3/detection_loss‚2.571147552558354 /loss/FRt1/detection_loss‚1.6857985043181822 .loss/FRt2/detection_loss‚2.509064549491519 0loss/SRqqt4/detection_loss‚1.750491916601147 _runtime‚4746.001935275 .loss/FRt3/detection_loss‚3.010215436590129 .loss/FBt2/detection_loss‚2.354764051735401 .loss/FBt4/detection_loss‚2.759894073009491‚î4wö‚(:… ®¥åÔ»¾õ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.25' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.25 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W~`» ÷‚(:› ®¥åÔĈ‘ disk.dm-0.in‚ 574.5546875$ memory_percent‚91.32049083709717 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚2.453125* proc.memory.percent‚17.531919479370117 disk.sda3.out‚ 186.2109375 disk.sda.in‚0 disk.sda1.in‚0" proc.memS±Íory.rssMB‚ 2872.4296875) disk./.usagePercent‚73.87532715714815 disk.sda2.out‚0 disk.rbd2.out‚ 141.001953125" disk.dm-0.out‚28087.2431640625 disk.sda.out‚ 186.2109375 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚28087.2431640625! disk.nvme0c0n1.in‚ 574.5546875% disk./.usageGB‚1100.7957611083984 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 cpu‚25.35900421025488 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 186.2109375 disk.nvme0n1.in‚ 574.5546875) proc.memory.availableMB‚ 1422.05078125 proc.cpu.threads‚17 network.sent‚ 1.4286273e+07 network.recv‚ 1.732394e+06 disk.rbd3.out‚0' disk.nvme0c0n1.out‚28087.24316406254à­‘ø‚(:† ½¥åÔ¥õì gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚91.365' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.273 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h@0qù‚(:‘ ½¥åÔëù° network.sent‚ 1.4395133e+07 disk.dm-1.in‚0 disk.sda2.out‚0 disk.rbd2.out‚ 141.064453125 disk.sda2.in‚0 disk.nvme0n1.in‚ 574.5546875 disk.dm-0.in‚ 574.5546875$ memory_percent‚91.32153987884521 disk.dm-1.out‚ 186.84375 disk.rbd3.in‚0 disk.sda1.in‚0) disk./.usagePercent‚73.87532357304268! disk.nvme0c0n1.in‚ 574.5546875 disk.sda.out‚ 186.84375 cpu‚25.358045215123177 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 network.recv‚ 1.7389e+06 disk.rbd1.out‚2.453125 disk.sda3.out‚ 186.84375 disk.drbd1777.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚28186.458984375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2872.4296875% disk./.usageGB‚1100.7957077026367$ disk.nvme0n1.out‚28186.458984375& disk.nvme0c0n1.out‚28186.458984375 disk.rbd2.in‚0) proc.memory.availableMB‚ 1421.87890625* proc.memory.percent‚17.531919479370117 proc.cpu.threads‚178ªøy2ú- _wandbruntime‚4780  _runtime‚4780À5µ'ÇûÁ & lr-AdamW/pg1‚0.0001247586539123369 & lr-AdamW/pg2‚0.0001247586539123369 trainer/global_step‚6299 % _timestamp‚1.7884330955987804e+09 _runtime‚4780.303522883 _step‚5158³BdÌüÆ & lr-AdamW/pg1‚0.0001247586539123369 & lr-AdamW/pg2‚0.0001247586539123369 trainer/global_step‚6299 % _timestamp‚1.7884330955987804e+09 _runtime‚4780.303522883 _step‚515ƒÃk’c2ý- _wandbruntime‚4780  _runtime‚4780ô²oá'þÛO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83712591 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 % _timestamp‚1.7884330955995042e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371921558765568 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362289534806418 _step‚516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30199729 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5538649807872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5259623174166 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366642498297344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4780.304447625 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91712286 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚371922233131008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210699067 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5279060468224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83712153 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0I¬º¤æ'ÿàO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362289534806418 _step‚516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30199729 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5538649807872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5259623174166 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366642498297344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4780.304447625 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91712286 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚371922233131008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210699067 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5279060468224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83712153 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83712591 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 % _timestamp‚1.7884330955995042e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371921558765568 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0„lj²]2€- _wandbruntime‚4780  _runtime‚4780S=ø%åM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5261094047094 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362344583894826 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5280536271872 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚400412809385984 % _timestamp‚1.7884330962557416e+09 _runtime‚4780.961120457 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 trainer/global_step‚6299 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5280536271872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91726302 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119025299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53521028 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5279879896064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362345255372150 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91726304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚371979405611008 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚366698869339136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚366698851336192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83724750 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405952130734592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53521030 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚400412785445888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119025299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119005235 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91726304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366698869339136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210751603 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚371979405611008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210751603 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30204156 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30203722 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405952322479616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5279879896064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83725186 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357084144285696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5539513093632 ]KDeviceStaôz]ÛtsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5539345288704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210731537 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119005235ÙÀEð&‚êM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚366698851336192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚371978731232256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 trainer/global_step‚6299 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5280536271872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91726302 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119025299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53521028 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5279879896064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362345255372150 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91726304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚371979405611008 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚366698869339136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚366698851336192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83724750 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405952130734592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53521030 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚400412785445888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119025299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119005235 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91726304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366698869339136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210751603 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚371979405611008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210751603 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30204156 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30203722 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405952322479616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5279879896064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83725186 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357084144285696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5539513093632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5539345288704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210731537 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119005235 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5261094047094 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362344583894826 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5280536271872 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚400412809385984 % _timestamp‚1.7884330962557416e+09 _runtime‚4780.961120457 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5301112951330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚369460150239744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92413088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369460168242688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5320702636544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5300458513574 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92413086 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚369460150239744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369460168242688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5581239768576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884331307074182e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92413086 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5320046274048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365075576673826 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119932871 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403433695531520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚365074905196710 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30424696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403433671591424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84345956 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚359774463722496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119932871 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374780196513792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 _step‚520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374780196513792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5320702636544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53921262 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409015103118336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440ÍH¼‘ßO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374780196513792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5320702636544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53921262 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409015103118336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409014911360000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 trainer/global_step‚6349 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚359774446683136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5320046274048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pSÒûŒ)ool.allocated‚ 5301112951330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚369460150239744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92413088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369460168242688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5320702636544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5300458513574 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92413086 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚369460150239744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369460168242688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5581239768576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884331307074182e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92413086 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5320046274048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365075576673826 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119932871 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403433695531520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚365074905196710 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30424696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403433671591424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84345956 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚359774463722496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119932871 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374780196513792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 _step‚520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ˆJ¸ Ñ2’- _wandbruntime‚4816  _runtime‚4816T+„Jë&“åM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119931287 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84358607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚369516521281536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212378455 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374838043359232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359829367853056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53929428 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212378455 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92427102 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884331313666875e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212358389 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚4816.071923581 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚365130625762234 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92427102 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84359051 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30429621 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5301929386502 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5320865701888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚6349 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409077429504000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119951351 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374838043359232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92427104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359829350813696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369516503278592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403495346269184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119931287 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5301274948538 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212358389 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5320865701888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369516521281536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30429179 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚409077612860416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5582266591232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53929430 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92427104 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚365130625762234 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92427102 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84359051 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30429621 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5301929386502 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5320865701888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚6349 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409077429504000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119951351 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374838043359232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92427104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359829350813696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369516503278592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403495346269184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119931287 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5301274948538 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212358389 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5320865701888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369516521281536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30429179 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚409077612860416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5582266591232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53929430 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92427104 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84358607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚369516521281536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212378455 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374838043359232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359829367853056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53929428 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212378455 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92427102 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884331313666875e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212358389 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚4816.071923581‰±yÊË2•- _wandbruntime‚4816  _runtime‚4816¯œ$]–× 2loss/SRqqt3/assignment_loss‚1.9598593673535756 0loss/FBt3/assignment_loss‚0.1308818417150178 1loss/SRqqt4/detection_loss‚1.6557603891728059 /loss/FBt1/detection_loss‚0.5381172398241917 epoch‚5 0loss/FRt4/assignment_loss‚2.9741328032511585 2loss/SRqqt1/assignment_loss‚0.8421544597691132 1loss/FBt2/assignment_loss‚0.23762597560926224 /loss/FRt4/detection_loss‚2.1892613251254245 0loss/SRqqt2/detection_loss‚2.068561255222275 .loss/FBt3/detection_loss‚2.942247474193573 0loss/FRt2/assignment_loss‚3.4780639880581905 2loss/SRqqt2/assignment_loss‚1.5051296428511185 1loss/FBt4/assignment_loss‚0.06466972322336265 .loss/FBt4/detection_loss‚3.252116935593741 trainer/global_step‚6349 0loss/FRt1/assignment_loss‚1.4970630587271925 2loss/SRqqt4/assignment_loss‚1.4169944015876301 1loss/FBt1/assignment_loss‚0.12932367081818444 1loss/SRqqt1/detection_loss‚0.8765689990379744 % _timestamp‚1.7884331313682115e+09 /loss/FRt3/assignment_loss‚3.237753125337454 .loss/FRt3/detection_loss‚3.080472783400462 _runtime‚4816.072832122 &loss/total_loss‚42.932793119323634 /loss/FRt1/detection_loss‚1.7587492817415382 /loss/FRt2/detection_loss‚2.5880966217894303 _step‚522 0loss/SRqqt3/detection_loss‚2.208100141797747 /loss/FBt2/detection_loss‚2.3010886141232083¤R—b—Ü 2loss/SRqqt3/assignment_loss‚1.9598593673535756 0loss/FBt3/assignment_loss‚0.1308818417150178 1loss/SRqqt4/detection_loss‚1.6557603891728059 /loss/FBt1/detection_loss‚0.5381172398241917 epoch‚5 0loss/FRt4/assignment_loss‚2.9741328032511585 2loss/SRqqt1/assignment_loss‚0.8421544597691132 1loss/FBt2/assignment_loss‚0.23762597560926224 /loss/FRt4/detection_loss‚2.1892613251254245 0loss/SRqqt2/detection_loss‚2.068561255222275 .loss/FBt3/detection_loss‚2.942247474193573 0loss/FRt2/assignment_loss‚3.4780639880581905 2loss/SRqqt2/assignment_loss‚1.5051296428511185 1loss/FBt4/assignment_loss‚0.06466972322336265 .loss/FBt4/detection_loss‚3.252116935593741 trainer/global_step‚6349 0loss/FRt1/assignment_loss‚1.4970630587271925 2loss/SRqqt4/assignment_loss‚1.4169944015876301 1loss/FBt1/assignment_loss‚0.12932367081818444 1loss/SRqqt1/detection_loss‚0.8765689990379744 % _timestamp‚1.7884331313682115e+09 /loss/FRt3/assignment_loss‚3.237753125337454 .loss/FRt3/detection_loss‚3.080472783400462 _runtime‚4816.072832122 &loss/total_loss‚42.932793119323634 /loss/FRt1/detection_loss‚1.7587492817415382 /loss/FRt2/detection_loss‚2.5880966217894303 _step‚522 0loss/SRqqt3/detection_loss‚2.208100141797747 /loss/FBt2/detection_loss‚2.3010886141232083ŠÃ,ãý“˜‚(:ˆ ù¥åÔ¦òÞ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.939' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.1878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v¥$, ™‚(:• ù¥åÔｿ disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 188.82421875 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 574.5546875! disk.rbd2.out‚142.83447265625$ memory_percent‚94.20642852783203% proc.memory.availableMB‚ 949.21875 disk.rbd3.out‚0* proc.memory.percent‚17.531919479370117$ disk./.usageGB‚1100.795669555664 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 574.5546875 disk.rbd0.in‚0 network.recv‚ 1.750645e+06) disk./.usagePercent‚73.87532101296733 disk.sda2.in‚0" proc.memory.rssMB‚ 2872.4296875 network.sent‚ 1.4514315e+07 disk.sda3.out‚ 188.82421875 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 188.82421875 disk.rbd1.out‚2.453125% disk.nvme0c0n1.out‚28484.80859375 disk.dm-0.out‚28484.80859375 disk.sda1.out‚0 disk.nvme0n1.in‚ 574.5546875# disk.nvme0n1.out‚28484.80859375 disk.sda2.out‚0 cpu‚25.35413480483913 proc.cpu.threads‚17š,>žš‚(:“ ˆ¦åÔϼõ gpu.0.gpu‚16.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.706599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,Âê$›‚(:™ ˆ¦åÔò®Ž‘ disk.nvme0n1.in‚ 574.5546875& disk.nvme0n1.out‚28584.29541015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚0( disk.nvme0c0n1.out‚28584.29541015625 disk.sda3.out‚ 189.359375$ memory_percent‚94.20738220214844 network.sent‚ 1.4621627e+07 disk.drbd1777.out‚0 disk.rbd2.in‚0$ proc.memory.availableMB‚949.0625 network.recv‚ 1.755973e+06) disk./.usagePercent‚73.87532920520843% disk./.usageGB‚1100.7957916259766 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 189.359375 disk.dm-1.out‚ 189.359375 disk.rbd1.in‚0" proc.memory.rssMB‚ 2872.4296875 disk.rbd3.in‚0 disk.rbd1.out‚2.453125! disk.nvme0c0n1.in‚ 574.5546875* proc.memory.percent‚17.531919479370117# disk.dm-0.out‚28584.29541015625 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.rbd2.out‚142.85009765625 cpu‚25.353243080956553 disk.sda1.out‚0 disk.dm-0.in‚ 574.5546875ã þ2œ- _wandbruntime‚4850  _runtime‚4850WµjÈ $ _timestamp‚1.788433165496093e+09 _runtime‚4850.200867301 _step‚523 ' lr-AdamW/pg1‚0.00012149951430559869 ' lr-AdamW/pg2‚0.00012149951430559869 trainer/global_step‚6399.(nÍžÇ ' lr-AdamW/pg2‚0.00012149951430559869 trainer/global_step‚6399 $ _timestamp‚1.788433165496093e+09 _runtime‚4850.200867301 _step‚523 ' lr-AdamW/pg1‚0.00012149951430559869‹—µ9´2Ÿ- _wandbruntime‚4850  _runtime‚4850‰ì†¬û ÚO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93113886 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120858923 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMÄ’îå%onitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412140551652352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93113888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213972811 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5341948290738 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5341293852982 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372277802182144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120858923 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54329662 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120838885 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚367860947064118 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚524 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚6399 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚377638834262016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54329660 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚367861618541234 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372277820185088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5624164002304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5361032079872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377639508627456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372277820185088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377638834262016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5361688442368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5361688442368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30649956 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚362519653211136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 $ _timestamp‚1.788433165496831e+09p5·3å'¡ßO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5361688442368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30649956 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚362519653211136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 $ _timestamp‚1.788433165496831e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93113886 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120858923 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412140551652352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93113888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213972811 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5341948290738 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5341293852982 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372277802182144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120858923 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54329662 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120838885 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚367860947064118 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚524 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚6399 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚377638834262016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54329660 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚367861618541234 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372277820185088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5624164002304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5361032079872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377639508627456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372277820185088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377638834262016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5361688442368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412203067332608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93127902 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372334173223936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚377696006728704 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120857339 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93127902 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372334155220992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5362507883520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5361851507712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚367916667629642 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372334173223936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5342764725910 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84992245 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362574557341696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214005307 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120857339 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54337828 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30654860 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 213985241 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5625184180224 _step‚525 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚362574574381056 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372334155220992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93127904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54337830 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377696681107456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93127904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214005307 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4850.858511966 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808+ºÎó÷ ¤éM $ _timestamp‚1.788433166153343e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30654417 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213985241 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120877403 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚6399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84992245 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362574557341696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214005307 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120857339 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54337828 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30654860 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 213985241 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5625184180224 _step‚525 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚362574574381056 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372334155220992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93127904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54337830 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377696681107456 XJDeviceStaMÐ÷øtsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93127904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214005307 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4850.858511966 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412202875587584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412203067332608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93127902 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372334173223936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚377696006728704 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120857339 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93127902 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372334155220992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5362507883520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5361851507712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚367916667629642 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372334173223936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5342764725910 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121784975 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121764937 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚380498146375680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380497472010240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365264859739136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚375095472127488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375095454124544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380497472010240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5402017885696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121764937 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85615027 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚415265836296704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚375095472127488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872}2‡#±àO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5667090938880 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93814686 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215599663 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93814688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5402674248192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121784975 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121764937 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚380498146375680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380497472010240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365264859739136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚375095472127488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375095454124544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380497472010240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5402017885696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121764937 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85615027 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚415265836296704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚375095472127488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30876519 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚6449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚409598745357824 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5667260854272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215579623 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚415266030152192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚375095454124544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_tþŒY_rain_batch_start/allocated_bytes.all.allocated‚380498146375680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370646988931526 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93814688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54738062 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚365264876778496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93814686 +äf2²- _wandbruntime‚4885  _runtime‚4885Ÿ?J$ë&³åM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚380555318855680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚529 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93828704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54746228 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5382945627354 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30881446 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5402837313536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚380555318855680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121803455 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215632159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54746230 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚375151825166336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365319780909056 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5402837313536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93828702 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121783391 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121803455 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93828702 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215612093 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375151825166336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚415328530833920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5403493689344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409660396095488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5667945090560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚370703380974374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4885.918516392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚380554644476928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5403493689344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85627226 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4885.918516392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚380554644476928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5403493689344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85627226 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5402837313536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚380555318855680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121803455 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215632159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54746230 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚375151825166336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365319780909056 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5402837313536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93828702 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121783391 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121803455 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93828702 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215612093 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375151825166336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚415328530833920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5403493689344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409660396095488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5667945090560‘ÛêŽ|2µ- _wandbruntime‚4885  _runtime‚4885ÊðZ¶Ô 2loss/SRqqt4/assignment_loss‚1.7160810103914776 .loss/FRt4/detection_loss‚2.181312308032462 0loss/SRqqt1/detection_loss‚1.336827969321838 /loss/FBt4/detection_loss‚2.8733437061309814 epoch‚5 trainer/global_step‚6449 _runtime‚4885.919573263 1loss/SRqqt2/detection_loss‚1.8014292494983088 &loss/total_loss‚43.820476393101245 0loss/FBt3/assignment_loss‚0.2987636514590122 0loss/FRt1/assignment_loss‚1.3830887754758199 0loss/FRt2/assignment_loss‚3.0938859052128262 0loss/FRt3/assignment_loss‚3.3924324115117392 2loss/SRqqt2/assignment_loss‚1.3814677121687908 .loss/FRt3/detection_loss‚3.429830346788679 /loss/FRt2/detection_loss‚2.2304930515863277 1loss/SRqqt3/detection_loss‚2.4599178810914357 0loss/SRqqt4/detection_loss‚1.565106717778034 /loss/FBt1/detection_loss‚0.5887243090424148 /loss/FBt3/detection_loss‚2.6328475922346115 1loss/FBt1/assignment_loss‚0.16783679129721712 .loss/FRt1/detection_loss‚1.836881395512157 0loss/FBt2/assignment_loss‚0.2690036806826746 1loss/FBt4/assignment_loss‚0.13146967782328525 .loss/FBt2/detection_loss‚3.156466313770839 /loss/FRt4/assignment_loss‚3.168312574954743 2loss/SRqqt1/assignment_loss‚0.8782262747319272 2loss/SRqqt3/assignment_loss‚1.8467270866036416 $ _timestamp‚1.788433201214971e+09 _step‚530k4ck_·Ù _runtime‚4885.919573263 1loss/SRqqt2/detection_loss‚1.8014292494983088 &loss/total_loss‚43.820476393101245 0loss/FBt3/assignment_loss‚0.2987636514590122 0loss/FRt1/assignment_loss‚1.3830887754758199 0loss/FRt2/assignment_loss‚3.0938859052128262 0loss/FRt3/assignment_loss‚3.3924324115117392 2loss/SRqqt2/assignment_loss‚1.3814677121687908 .loss/FRt3/detection_loss‚3.429830346788679 /loss/FRt2/detection_loss‚2.2304930515863277 1loss/SRqqt3/detection_loss‚2.4599178810914357 0loss/SRqqt4/detection_loss‚1.565106717778034 /loss/FBt1/detection_loss‚0.5887243090424148 /loss/FBt3/detection_loss‚2.6328475922346115 1loss/FBt1/assignment_loss‚0.16783679129721712 .loss/FRt1/detection_loss‚1.836881395512157 0loss/FBt2/assignment_loss‚0.2690036806826746 1loss/FBt4/assignment_loss‚0.13146967782328525 .loss/FBt2/detection_loss‚3.156466313770839 /loss/FRt4/assignment_loss‚3.168312574954743 2loss/SRqqt1/assignment_loss‚0.8782262747319272 2loss/SRqqt3/assignment_loss‚1.8467270866036416 $ _timestamp‚1.788433201214971e+09 _step‚530 2loss/SRqqt4/assignment_loss‚1.7160810103914776 .loss/FRt4/detection_loss‚2.181312308032462 0loss/SRqqt1/detection_loss‚1.336827969321838 /loss/FBt4/detection_loss‚2.8733437061309814 epoch‚5 trainer/global_step‚6449’•<÷¸‚(:’ µ¦åÔžóÒ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.45' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.889999999999993 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÅ­=,¹‚(:¡ µ¦åÔÚûã$ disk./.usageGB‚1100.796085357666 disk.rbd0.out‚0 disk.sda3.out‚ 190.82421875 disk.dm-0.in‚ 574.6171875 disk.rbd1.out‚2.453125( proc.memory.availableMB‚ 948.76171875* proc.memory.percent‚17.531919479370117 proc.cpu.threads‚17 network.recv‚ 1.767607e+06 disk.dm-1.out‚ 190.82421875 disk.rbd2.in‚0 disk.rbd2.out‚143.0146484375 disk.sda1.out‚0 disk.nvme0n1.in‚ 574.6171875 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 574.6171875$ memory_percent‚94.20921802520752 network.sent‚ 1.4739328e+07) disk./.usagePercent‚73.87534891778856 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 cpu‚25.350369006576727# disk.dm-0.out‚28783.63623046875 disk.sda.out‚ 190.82421875 disk.dm-1.in‚0& disk.nvme0n1.out‚28783.63623046875 disk.sda2.in‚0( disk.nvme0c0n1.out‚28783.63623046875" proc.memory.rssMB‚ 2872.4296875ö±Œ¿“º‚(:ˆ ĦåÔÉ´å gpu.0.gpu‚50.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢,õ¢&»‚(:› ĦåÔ‹Ä£‘ disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚2.453125 disk.rbd0.out‚0$ memory_percent‚94.20859813690186( proc.memory.availableMB‚ 948.86328125 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda.out‚ 191.421875 disk.sda1.out‚0" proc.memory.rssMB‚ 2872.4296875% disk./.usageGB‚1100.7966957092285( disk.nvme0c0n1.out‚28883.63916015625 disk.sda2.in‚0 disk.dm-1.out‚ 191.421875 network.sent‚ 1.4846205e+07) disk./.usagePercent‚73.87538987899404 disk.rbd3.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.531919479370117 disk.sda3.out‚ 191.421875& disk.nvme0n1.out‚28883.63916015625 disk.rbd2.in‚0 cpu‚25.34945000191411 disk.rbd2.out‚144.5419921875 network.recv‚ 1.772231e+06 disk.dm-0.in‚ 574.6171875# disk.dm-0.out‚28883.63916015625 disk.sda.in‚0 disk.nvme0n1.in‚ 574.6171875 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 574.6171875 disk.sda2.out‚0 disk.sda3.in‚0U4 U¼‚(:’ Ó¦åÔ»óì gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.51' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.101999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ#M2'½‚(:œ Ó¦åÔ«ð„‘ disk.sda3.out‚ 192.01953125* proc.memory.percent‚17.531919479370117 disk.sda2.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚28883.93603515625 disk.rbd1.in‚0 disk.rbd1.out‚2.453125 disk.sda3.in‚0( disk.nvme0c0n1.out‚28883.93603515625" proc.memory.rssMB‚ 2872.4296875 network.recv‚ 1.774589e+06% disk./.usageGB‚1100.7967262268066 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 192.01953125 network.sent‚ 1.4849971e+07) disk./.usagePercent‚73.87539192705432 disk.rbd2.out‚144.6044921875 disk.sda.out‚ 192.01953125 disk.sda1.out‚0 cpu‚25.34854433126659 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 574.6171875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd3.in‚0# disk.dm-0.out‚28883.93603515625 disk.sda.in‚0 disk.nvme0n1.in‚ 574.6171875! disk.nvme0c0n1.in‚ 574.6171875# proc.memory.availableMB‚949.125 disk.sda1.in‚0 disk.dm-1.in‚0$ memory_percent‚94.20700073242188 disk.rbd0.in‚0W:#á2¾- _wandbruntime‚4920  _runtime‚4920ăÉ¿Ã % _timestamp‚1.7884332353126473e+09 _runtime‚4920.017315112 _step‚531 ' lr-AdamW/pg1‚0.00011823354949427389 ' lr-AdamW/pg2‚0.00011823354949427389 trainer/global_step‚6499¦#VõÎÀÈ % _timestamp‚1.7884332353126473e+09 _runtime‚4920.017315112 _step‚531 ' lr-AdamW/pg1‚0.00011823354949427389 ' lr-AdamW/pg2‚0.00011823354949427389 trainer/global_step‚6499“Cs'2Á- _wandbruntime‚4920  _runtime‚4920H¶%êÂÚO BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚383356109758464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412681306181120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122690989 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55146462 _step‚532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217226515 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217206475 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94515486 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122690989 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94515488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86250805 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31103877 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytL'‡öes.large_pool.freed‚412681282241024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31104343 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5710219687424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86250337 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5422964531798 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚383356109758464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377913124069888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373433702276050 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5423618969554 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418391525868544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚383356784123904 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122711027 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377913106066944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377913124069888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377913106066944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94515488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86250805 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31103877 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚412681282241024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31104343 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5710219687424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86250337 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5422964531798 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚383356109758464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377913124069888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373433702276050 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5423618969554 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418391525868544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚383356784123904 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122711027 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377913106066944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377913124069888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377913106066944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚383356109758464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412681306181120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122690989 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55146462 _step‚532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217226515 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217206475 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94515486 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122690989 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944”ßØø2Ä- _wandbruntime‚4920  _runtime‚4920Tw°·ê&ÅäM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373488751364458 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚383413956603904 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94529502 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94529502 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5444479495168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5443823119360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377969477108736 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217238945 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94529504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217238945 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412742956918784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217259011 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86263151 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 $ _timestamp‚1.788433235970333e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122709443 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86263601 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5443823119360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122709443 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418454033107968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55154630 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418453853945856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚6499 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122729507 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5424435404726 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377969477108736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377969459105792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5710920967168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31108971 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚4920.675517022 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31108523 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚373489422841782ÐM³¼æÆéM PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5424435404726 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377969477108736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377969459105792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5710920967168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31108971 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚4920.675517022 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31108523 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚373489422841782 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373488751364458 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚383413956603904 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94529502 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94529502 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5444479495168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5443823119360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377969477108736 B+E) nd/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217238945 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94529504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217238945 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412742956918784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217259011 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86263151 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 $ _timestamp‚1.788433235970333e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122709443 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86263601 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5443823119360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122709443 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418454033107968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55154630 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418453853945856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚6499 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122729507 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0•«aA²2Ç- _wandbruntime‚4920  _runtime‚4920ª¼h”\ÈÖ /loss/FRt1/assignment_loss‚2.036624690348452 1loss/SRqqt4/assignment_loss‚1.690050233090133 0loss/SRqqt4/detection_loss‚1.722779695402112 /loss/FBt2/detection_loss‚3.7379416624704995 0loss/FRt4/assignment_loss‚3.6621074096576587 2loss/SRqqt1/assignment_loss‚1.1977322277042173 1loss/SRqqt3/assignment_loss‚1.977928444561435 /loss/FBt3/detection_loss‚1.9130935743451118 epoch‚5 trainer/global_step‚6499 0loss/FRt3/assignment_loss‚2.5141827338620235 2loss/SRqqt2/assignment_loss‚1.5932978620683704 % _timestamp‚1.7884332359719908e+09 _runtime‚ 4920.67664677 _step‚534 1loss/FBt3/assignment_loss‚0.22784847143338993 .loss/FBt4/detection_loss‚2.285331144928932 .loss/FRt2/detection_loss‚2.518520411517885 1loss/SRqqt2/detection_loss‚1.6827783932288487 1loss/SRqqt3/detection_loss‚2.3186447518627817 1loss/FBt4/assignment_loss‚0.05665553292783443 1loss/SRqqt1/detection_loss‚1.1632767364142402 .loss/FBt1/detection_loss‚0.921997853069167 1loss/FBt1/assignment_loss‚0.10700283335290603 2loss/FBt2/assignment_loss‚0.017451866471674293 /loss/FRt3/detection_loss‚2.6367608089196053 /loss/FRt4/detection_loss‚2.6786021123061308 /loss/FRt2/assignment_loss‚3.019057830174764 /loss/FRt1/detection_loss‚1.6421804624524983 &loss/total_loss‚43.321847742570675õa‘aÉÛ /loss/FRt2/assignment_loss‚3.019057830174764 /loss/FRt1/detection_loss‚1.6421804624524983 &loss/total_loss‚43.321847742570675 /loss/FRt1/assignment_loss‚2.036624690348452 1loss/SRqqt4/assignment_loss‚1.690050233090133 0loss/SRqqt4/detection_loss‚1.722779695402112 /loss/FBt2/detection_loss‚3.7379416624704995 0loss/FRt4/assignment_loss‚3.6621074096576587 2loss/SRqqt1/assignment_loss‚1.1977322277042173 1loss/SRqqt3/assignment_loss‚1.977928444561435 /loss/FBt3/detection_loss‚1.9130935743451118 epoch‚5 trainer/global_step‚6499 0loss/FRt3/assignment_loss‚2.5141827338620235 2loss/SRqqt2/assignment_loss‚1.5932978620683704 % _timestamp‚1.7884332359719908e+09 _runtime‚ 4920.67664677 _step‚534 1loss/FBt3/assignment_loss‚0.22784847143338993 .loss/FBt4/detection_loss‚2.285331144928932 .loss/FRt2/detection_loss‚2.518520411517885 1loss/SRqqt2/detection_loss‚1.6827783932288487 1loss/SRqqt3/detection_loss‚2.3186447518627817 1loss/FBt4/assignment_loss‚0.05665553292783443 1loss/SRqqt1/detection_loss‚1.1632767364142402 .loss/FBt1/detection_loss‚0.921997853069167 1loss/FBt1/assignment_loss‚0.10700283335290603 2loss/FBt2/assignment_loss‚0.017451866471674293 /loss/FRt3/detection_loss‚2.6367608089196053 /loss/FRt4/detection_loss‚2.6786021123061308–5‚…“Ê‚(:ˆ â¦åÔœµãŽ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.086' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Š·B(Ë‚(: â¦åÔÛ ¯‘ disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 1.778562e+06) disk./.usagePercent‚73.87539448712967 disk.rbd1.out‚2.453125 disk.sda3.in‚0 disk.dm-0.in‚ 579.4609375 disk.sda.out‚ 192.4453125 disk.sda1.in‚0" proc.memory.rssMB‚ 2872.4296875 disk.drbd1777.out‚0 disk.sda.in‚0( disk.nvme0c0n1.out‚28983.40673828125 disk.drbd1777.in‚0 network.sent‚ 1.485529e+07% disk./.usageGB‚1100.7967643737793 disk.dm-1.out‚ 192.4453125 disk.nvme0n1.in‚ 579.4609375 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 579.4609375 disk.rbd2.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚28983.40673828125 disk.sda3.out‚ 192.4453125$ memory_percent‚94.20750141143799( proc.memory.availableMB‚ 949.04296875* proc.memory.percent‚17.531919479370117 proc.cpu.threads‚17 disk.rbd2.out‚144.6201171875 disk.rbd3.in‚0 cpu‚25.34763818071348 disk.dm-1.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚28983.40673828125㼓̂(:ˆ ñ¦åÔèßÀ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.774' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.7548 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûà `&Í‚(:› ñ¦åÔí°Ì disk.sda.out‚ 192.96875 network.recv‚ 1.783464e+06) disk./.usagePercent‚73.87539448712967# disk.dm-0.out‚29082.53955078125 disk.dm-1.out‚ 192.96875 disk.rbd1.in‚0( disk.nvme0c0n1.out‚29082.53955078125* proc.memory.percent‚17.531919479370117% disk./.usageGB‚1100.7967643737793 disk.dm-1.in‚0 disk.rbd1.out‚ 2.6015625 disk.drbd1777.out‚0$ memory_percent‚94.21055316925049 disk.sda1.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0( proc.memory.availableMB‚ 948.54296875 cpu‚25.346709442687146 disk.dm-0.in‚ 579.4609375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 192.96875 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0! disk.rbd2.out‚144.70654296875 disk.rbd3.out‚0 disk.nvme0n1.in‚ 579.4609375& disk.nvme0n1.out‚29082.53955078125! disk.nvme0c0n1.in‚ 579.4609375" proc.memory.rssMB‚ 2872.4296875 network.sent‚ 1.4962508e+07 disk.rbd3.in‚0 disk.rbd0.in‚0©¸á2Î- _wandbruntime‚4955  _runtime‚4955ûRkÇÏÁ & lr-AdamW/pg1‚0.0001165988582100729 & lr-AdamW/pg2‚0.0001165988582100729 trainer/global_step‚6549 % _timestamp‚1.7884332703110714e+09 _runtime‚4955.015858928 _step‚535îšÂÌÐÆ & lr-AdamW/pg1‚0.0001165988582100729 & lr-AdamW/pg2‚0.0001165988582100729 trainer/global_step‚6549 % _timestamp‚1.7884332703110714e+09 _runtime‚4955.015858928 _step‚535—1gŠÂ2Ñ- _wandbruntime‚4955  _runtime‚4955Ç•Xåà'ÒÚO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5484645859840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86885034 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123617041 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415763843064320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421516979107840 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218833327 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚6549 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31330611 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5752964030976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218853367 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5483989497344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86885473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31330174 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚380730776012288 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386214747506688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386214747506688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5753136043520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415763819124224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5464454308962 _step‚536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123637079 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55554862 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380730758009344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5484645859840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370755272795136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5463799871206 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55554860 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5483989497344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386215421872128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95216286 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95216288 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380730776012288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386215421872128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95216286 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚380730758009344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123617041 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376219072666342 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421516979107840 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218833327 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚6549 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31330611 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5752964030976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218853367 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5483989497344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86885473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31330174 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚380730776012288 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386214747506688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386214747506688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5753136043520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415763819124224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5464454308962 _step‚536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123637079 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55554862 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380730758009344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5484645859840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370755272795136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5463799871206 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55554860 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5483989497344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386215421872128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95216286 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95216288 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380730776012288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386215421872128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95216286 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚380730758009344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123617041 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376219072666342 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚370810176925696 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95230304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218865797 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31334616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386271919973376 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95230302 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218885863 S?DeviceStatsMonitor/Ë1µ.on_train_batch_end/requested_bytes.all.freed‚376274793231866 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6549 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380787111048192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415825493801984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31335049 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5485465300992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123635495 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370810193965056 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86897644 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5465270744134 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380787129051136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4955.668263537 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380787129051136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5484808925184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5484808925184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31335049 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5485465300992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123635495 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370810193965056 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86897644 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5465270744134 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380787129051136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4955.668263537 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380787129051136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5484808925184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5484808925184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚370810176925696 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95230304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218865797 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31334616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386271919973376 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95230302 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218885863 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚376274793231866 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6549 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380787111048192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415825493801984™Ù‰¸W2×- _wandbruntime‚4955  _runtime‚4955i XØÒ 0loss/FBt3/assignment_loss‚0.4386592766386457 /loss/FRt1/assignment_loss‚1.301918915162484 %loss/total_loss‚41.47575913696691 2loss/SRqqt2/assignment_loss‚1.3728860152118347 .loss/FRt1/detection_loss‚1.836541373282671 0loss/SRqqt2/detection_loss‚1.885960433120821 epoch‚5 _step‚538 2loss/SRqqt3/assignment_loss‚1.7105714424767278 1loss/FBt1/assignment_loss‚0.11367468479466891 /loss/FBt3/detection_loss‚2.5648914178212485 trainer/global_step‚6549 1loss/SRqqt4/assignment_loss‚1.706006399150622 /loss/FBt4/detection_loss‚2.1304718255996704 .loss/FRt4/assignment_loss‚3.15350700351927 1loss/FBt2/assignment_loss‚0.10658915776584763 /loss/FRt3/detection_loss‚2.9049562667355393 .loss/FRt4/detection_loss‚2.485416167312198 0loss/SRqqt3/detection_loss‚2.243142133409327 /loss/FBt1/detection_loss‚0.7054707262515356 1loss/FBt4/assignment_loss‚0.17355625270507657 0loss/SRqqt1/detection_loss‚1.063688148790207 1loss/SRqqt4/detection_loss‚1.7582817979283252 /loss/FBt2/detection_loss‚3.3325592994689943 _runtime‚4955.669598872 /loss/FRt2/detection_loss‚2.5080471070189225 0loss/FRt2/assignment_loss‚2.5058042783486214 0loss/FRt3/assignment_loss‚2.6005824425003747 2loss/SRqqt1/assignment_loss‚0.8725765719532692 $ _timestamp‚1.788433270964986e+09k Zš]Ù× 1loss/FBt1/assignment_loss‚0.11367468479466891 /loss/FBt3/detection_loss‚2.5648914178212485 trainer/global_step‚6549 1loss/SRqqt4/assignment_loss‚1.706006399150622 /loss/FBt4/detection_loss‚2.1304718255996704 .loss/FRt4/assignment_loss‚3.15350700351927 1loss/FBt2/assignment_loss‚0.10658915776584763 /loss/FRt3/detection_loss‚2.9049562667355393 .loss/FRt4/detection_loss‚2.485416167312198 0loss/SRqqt3/detection_loss‚2.243142133409327 /loss/FBt1/detection_loss‚0.7054707262515356 1loss/FBt4/assignment_loss‚0.17355625270507657 0loss/SRqqt1/detection_loss‚1.063688148790207 1loss/SRqqt4/detection_loss‚1.7582817979283252 /loss/FBt2/detection_loss‚3.3325592994689943 _runtime‚4955.669598872 /loss/FRt2/detection_loss‚2.5080471070189225 0loss/FRt2/assignment_loss‚2.5058042783486214 0loss/FRt3/assignment_loss‚2.6005824425003747 2loss/SRqqt1/assignment_loss‚0.8725765719532692 $ _timestamp‚1.788433270964986e+09 0loss/FBt3/assignment_loss‚0.4386592766386457 /loss/FRt1/assignment_loss‚1.301918915162484 %loss/total_loss‚41.47575913696691 2loss/SRqqt2/assignment_loss‚1.3728860152118347 .loss/FRt1/detection_loss‚1.836541373282671 0loss/SRqqt2/detection_loss‚1.885960433120821 epoch‚5 _step‚538 2loss/SRqqt3/assignment_loss‚1.7105714424767278š{?5DžÚ‚(:“ €§åÔþÄú gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚135.246' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.049200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÛC&Û‚(:› €§åÔûà‘) disk./.usagePercent‚73.87537835865501 disk.nvme0n1.in‚ 579.4609375! disk.rbd2.out‚144.72216796875 disk.sda3.out‚ 193.5859375 disk.drbd1777.out‚0$ memory_percent‚94.21069622039795 network.sent‚ 1.4968001e+07 disk.rbd2.in‚0$ disk.nvme0n1.out‚29181.966796875 disk.dm-0.in‚ 579.4609375 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 193.5859375 disk.rbd1.out‚ 2.6015625 disk.sda2.out‚0( proc.memory.availableMB‚ 948.51953125" proc.memory.rssMB‚ 2872.4296875* proc.memory.percent‚17.531919479370117 proc.cpu.threads‚17 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 579.4609375 disk.rbd0.out‚0% disk./.usageGB‚1100.7965240478516& disk.nvme0c0n1.out‚29181.966796875 disk.sda3.in‚0 disk.drbd1777.in‚0 network.recv‚ 1.787606e+06! disk.dm-0.out‚29181.966796875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 193.5859375 disk.rbd3.in‚0 cpu‚25.345802845983783?ÙÄ“Ü‚(:ˆ §åÔý•Ú gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚153.974' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7948 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºJ,0Ý‚(:¥ §åÔ³Øá& disk.nvme0n1.out‚29281.07275390625( disk.nvme0c0n1.out‚29281.07275390625 disk.rbd0.out‚0" proc.memory.rssMB‚ 2872.4296875 disk.sda3.out‚ 194.33984375 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 579.4609375$ memory_percent‚94.20690536499023 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 194.33984375 network.sent‚ 1.5075644e+07 disk.rbd1.out‚ 2.6337890625 disk.sda2.in‚0 disk.sda2.out‚0! disk.rbd2.out‚144.78466796875* proc.memory.percent‚17.531919479370117 disk.dm-1.out‚ 194.33984375! disk.nvme0c0n1.in‚ 579.4609375& proc.memory.availableMB‚ 949.140625 cpu‚25.344879620063836 disk.rbd0.in‚0% disk./.usageGB‚1100.7965240478516 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚29281.07275390625 disk.rbd1.in‚0 disk.dm-0.in‚ 579.4609375 network.recv‚ 1.792485e+06) disk./.usagePercent‚73.87537835865501 proc.cpu.threads‚17 Jø¾2Þ- _wandbruntime‚4989  _runtime‚4989-¿‚0Éßà % _timestamp‚1.7884333048615007e+09 _runtime‚4989.566249548 _step‚539 ' lr-AdamW/pg1‚0.00011496348182520802 ' lr-AdamW/pg2‚0.00011496348182520802 trainer/global_step‚6599b¹DÎàÈ ' lr-AdamW/pg1‚0.00011496348182520802 ' lr-AdamW/pg2‚0.00011496348182520802 trainer/global_step‚6599 % _timestamp‚1.7884333048615007e+09 _runtime‚4989.566249548 _step‚539›)„Äw2á- _wandbruntime‚4989  _runtime‚4989¦œÞÐ!âÛO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5505289648370 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚389073385254912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5795919723520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379005114533750 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5525631665664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424642459100672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124543093 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5796079153152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5524975303168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚540 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87521910 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418846379947520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95917086 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389074059620352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31558650 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373500479323136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124563131 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚4989.567252788 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124543093 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚383548427954688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87522363 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚383548409951744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124563131 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379005786010866 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚6599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220460179 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55963262 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884333048622453e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31559101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95917088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_K™¬ôQpool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95917086 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55963260 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389074059620352 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389073385254912 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124563131 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379005786010866 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚6599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220460179 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55963262 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884333048622453e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31559101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95917088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95917086 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55963260 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389074059620352 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389073385254912 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379005114533750 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5525631665664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424642459100672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124543093 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5796079153152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5524975303168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚540 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87521910 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418846379947520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95917086 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389074059620352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31558650 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373500479323136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124563131 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚4989.567252788 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124543093 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30œ:úί2ä- _wandbruntime‚4990  _runtime‚4990Ãj/{ê&åäM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚418908030685184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379061506576598 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5525794731008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220492649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220512715 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124561547 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373555383453696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95931104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389131232100352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚383604762990592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55971430 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87535077 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5505451645578 trainer/global_step‚6599 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87534624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚383604780993536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220512715 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124581611 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5526451106816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚389130557721600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚389131232100352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424704782787072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373555400493056 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5525794731008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424704972434944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚541 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220492649 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95931102 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124581611 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55971428 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5796776041984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚383604780993536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31563196 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚4990.431903713 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95931102 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096ü7‰6ï&æéM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚418908030685184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379061506576598 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5525794731008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220492649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220512715 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124561547 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373555383453696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95931104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389131232100352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚383604762990592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55971430 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87535077 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5505451645578 trainer/global_step‚6599 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87534624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚383604780993536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220512715 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124581611 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5526451106816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚389130557721600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚389131232100352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424704782787072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373555400493056 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5525794731008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424704972434944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚541 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220492649 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95931102 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124581611 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55971428 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5796776041984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚383604780993536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31563196 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚4990.431903713 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95931102 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096NWå2ç- _wandbruntime‚4990  _runtime‚4990ÁKÞf[èÚ /loss/FBt1/detection_loss‚0.6964186039411199 2loss/SRqqt2/assignment_loss‚1.5598235310891928 .loss/FRt3/detection_loss‚2.886870764916943 .loss/FBt3/detection_loss‚3.601269245147705 2loss/SRqqt3/assignment_loss‚1.5623949327402644 2loss/FBt3/assignment_loss‚0.024862037417581934 % _timestamp‚1.7884333057283175e+09 .loss/FRt4/detection_loss‚2.196649816690707 1loss/SRqqt1/detection_loss‚1.1167222324875183 1loss/SRqqt3/detection_loss‚2.2606460783216686 /loss/FBt2/detection_loss‚2.2314757456382117 _runtime‚4990.432959362 0loss/FRt1/assignment_loss‚1.3442913249135018 /loss/FRt3/assignment_loss‚2.773931545595969 2loss/SRqqt1/assignment_loss‚0.7963289168837946 /loss/FRt1/detection_loss‚2.0882281532045455 1loss/SRqqt4/detection_loss‚1.8820874355733395 0loss/FRt2/assignment_loss‚2.Sw:W5707419857382776 2loss/SRqqt4/assignment_loss‚1.6219311146331685 1loss/SRqqt2/detection_loss‚1.9034670566098164 _step‚542 1loss/FBt1/assignment_loss‚0.06936854127243963 1loss/FBt2/assignment_loss‚0.16150676731725375 %loss/total_loss‚42.08361692751481 epoch‚5 trainer/global_step‚6599 /loss/FBt4/detection_loss‚2.7427222984177724 0loss/FRt4/assignment_loss‚3.3784167836694157 1loss/FBt4/assignment_loss‚0.10246188645916325 /loss/FRt2/detection_loss‚2.5110001288354398€Ù•ceéß 2loss/SRqqt3/assignment_loss‚1.5623949327402644 2loss/FBt3/assignment_loss‚0.024862037417581934 % _timestamp‚1.7884333057283175e+09 .loss/FRt4/detection_loss‚2.196649816690707 1loss/SRqqt1/detection_loss‚1.1167222324875183 1loss/SRqqt3/detection_loss‚2.2606460783216686 /loss/FBt2/detection_loss‚2.2314757456382117 _runtime‚4990.432959362 0loss/FRt1/assignment_loss‚1.3442913249135018 /loss/FRt3/assignment_loss‚2.773931545595969 2loss/SRqqt1/assignment_loss‚0.7963289168837946 /loss/FRt1/detection_loss‚2.0882281532045455 1loss/SRqqt4/detection_loss‚1.8820874355733395 0loss/FRt2/assignment_loss‚2.5707419857382776 2loss/SRqqt4/assignment_loss‚1.6219311146331685 1loss/SRqqt2/detection_loss‚1.9034670566098164 _step‚542 1loss/FBt1/assignment_loss‚0.06936854127243963 1loss/FBt2/assignment_loss‚0.16150676731725375 %loss/total_loss‚42.08361692751481 epoch‚5 trainer/global_step‚6599 /loss/FBt4/detection_loss‚2.7427222984177724 0loss/FRt4/assignment_loss‚3.3784167836694157 1loss/FBt4/assignment_loss‚0.10246188645916325 /loss/FRt2/detection_loss‚2.5110001288354398 /loss/FBt1/detection_loss‚0.6964186039411199 2loss/SRqqt2/assignment_loss‚1.5598235310891928 .loss/FRt3/detection_loss‚2.886870764916943 .loss/FBt3/detection_loss‚3.601269245147705ž#ßû3“ê‚(:ˆ ž§åÔö›ð gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.141' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.6282 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³¦mœ(ë‚(: ž§åÔȤ”‘ network.sent‚ 1.5078381e+07 network.recv‚ 1.794238e+06 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 194.828125 disk.dm-1.out‚ 194.828125 disk.sda1.out‚0* proc.memory.percent‚17.531919479370117) disk./.usagePercent‚73.87538219876801 disk.rbd2.in‚0 disk.dm-0.in‚ 579.4609375 disk.sda1.in‚0' disk.nvme0c0n1.out‚29281.8056640625 cpu‚25.343984111823655 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 194.828125 disk.rbd3.in‚0 disk.dm-1.in‚0% disk./.usageGB‚1100.7965812683105 disk.rbd3.out‚0! disk.nvme0c0n1.in‚ 579.4609375" proc.memory.rssMB‚ 2872.4296875 disk.drbd1777.out‚0" disk.dm-0.out‚29281.8056640625 disk.nvme0n1.in‚ 579.4609375 disk.rbd1.out‚ 2.6337890625 disk.drbd1777.in‚0% disk.nvme0n1.out‚29281.8056640625 disk.rbd1.in‚0$ memory_percent‚94.20723915100098' proc.memory.availableMB‚ 949.0859375 disk.rbd0.out‚0! disk.rbd2.out‚146.31201171875 proc.cpu.threads‚17šHžì‚(:“ ­§åÔ‘€Ž gpu.0.gpu‚33.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.967' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.393400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇZG,í‚(:¡ ­§åÔ°œ± disk.drbd1777.in‚0% disk.nvme0n1.out‚29380.9580078125 network.sent‚ 1.518734e+07 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 195.51171875$ memory_percent‚94.20797824859619( proc.memory.availableMB‚ 948.96484375' disk./.usagePercent‚73.875359670105 disk.sda.out‚ 195.51171875 disk.dm-1.in‚0 disk.sda3.out‚ 195.51171875 network.recv‚ 1.800858e+06% disk./.usageGB‚1100.7962455749512 disk.rbd3.in‚0 disk.dm-0.in‚ 579.4609375" disk.dm-0.out‚29380.9580078125 disk.sda1.in‚0" proc.memory.rssMB‚ 2872.4296875 disk.drbd1777.out‚0 cpu‚25.34306679041755 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 2.69677734375' disk.nvme0c0n1.out‚29380.9580078125 disk.nvme0n1.in‚ 579.4609375 disk.rbd1.in‚0* proc.memory.percent‚17.531919479370117 proc.cpu.threads‚17! disk.rbd2.out‚146.37451171875! disk.nvme0c0n1.in‚ 579.4609375 disk.sda3.in‚0£F½2î- _wandbruntime‚5024  _runtime‚5024æœøyÉïà _step‚543 ' lr-AdamW/pg1‚0.00011332776114966665 ' lr-AdamW/pg2‚0.00011332776114966665 trainer/global_step‚6649 % _timestamp‚1.7884333396808639e+09 _runtime‚5024.385605719PG•÷ÎðÈ ' lr-AdamW/pg1‚0.00011332776114966665 ' lr-AdamW/pg2‚0.00011332776114966665 trainer/global_step‚6649 % _timestamp‚1.7884333396808639e+09 _runtime‚5024.385605719 _step‚543Ÿ;67ž2ñ- _wandbruntime‚5024  _runtime‚5024lo´9á'òÛO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222087031 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5566617471488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96617886 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125489183 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88157858 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5024.386500324 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚391932023003136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376245702890496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376245685851136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391932023003136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96617886 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96617888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5565961108992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56371662 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386366061894144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125489183 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚391932697368576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚391932697368576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 58388705510407<;Òæ'óàO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5566617471488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56371662 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386366061894144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125489183 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚391932697368576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚391932697368576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5838870551040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222087031 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5566617471488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96617886 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125489183 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88157858 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5024.386500324 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚391932023003136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376245702890496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376245685851136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391932023003136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96617886 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96617888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5565961108992 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚423 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 trainer/global_step‚6649 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5566780536832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31790238 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5546941422950 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚5025.053789311 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376300607021056 % _timestamp‚1.7884333403486097e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5566780536832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222139567 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚386422414932992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31790661 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚391989195469824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5567436912640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421990567568384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125487599 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222119501 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56379828 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88170491 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222119501 EDeviceStatsMonitor.on_train_batch_end/inactive_split.”ÿ§gýall.freed‚88170066 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5839735248896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376300589981696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚427830464330752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125487599 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚425 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96631904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚386422414932992 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96631902 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5567436912640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391989869848576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚386422432935936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386422432935936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125507663 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5839896762368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5546286984986 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚427830278877184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚423 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 trainer/global_step‚6649 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5566780536832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31790238 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5546941422950 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚5025.053789311 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376300607021056 % _timestamp‚1.7884333403486097e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5566780536832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222139567 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚386422414932992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31790661 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚391989195469824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5567436912640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421990567568384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125487599 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222119501 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56379828 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88170491 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222119501 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88170066 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5839735248896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376300589981696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚427830464330752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125487599 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚425 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96631904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚386422414932992 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96631902 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5567436912640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391989869848576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚386422432935936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386422432935936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125507663 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5839896762368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5546286984986 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928¡õP°-2ù- _wandbruntime‚5025  _runtime‚5025³ØYVúÐ 2loss/SRqqt3/assignment_loss‚1.8613210434422773 /loss/SRqqt4/detection_loss‚1.91558374799788 .loss/FBt3/detection_loss‚2.891174920967647 epoch‚5 % _timestamp‚1.7884333403502083e+09 trainer/global_step‚6649 /loss/FRt2/assignment_loss‚3.245216030266977 /loss/FRt3/detection_loss‚3.4502933859825133 _runtime‚ 5025.05486093 0loss/FRt1/assignment_loss‚1.4644047247038947 0loss/FRt3/assignment_loss‚3.1450573444366454 0loss/FRt4/assignment_loss‚2.8955184671936967 /loss/FRt1/detection_loss‚1.5670505926012992 .loss/FRt4/detection_loss‚2.344895240737171 1loss/SRqqt1/detection_loss‚0.9885815089083698 .loss/FBt2/detection_loss‚2.337576389312744 /loss/FBt4/detection_loss‚2.5120398104190826 2loss/SRqqt2/assignment_loss‚1.6655458400398493 1loss/SRqqt2/detection_loss‚1.8615443593263625 0loss/SRqqt3/detection_loss‚2.347577988224871 2loss/SRqqt1/assignment_loss‚0.7647526080222571 1loss/FBt1/assignment_loss‚0.18376289788112102 1loss/FBt2/assignment_loss‚0.15154271781567433 0loss/FBt3/assignment_loss‚0.5795348157574024 /loss/FBt4/assignment_loss‚0.444189742517968 /loss/FBt1/detection_loss‚0.5954255768972472 _step‚546 1loss/SRqqt4/assignment_loss‚1.539214524673298 .loss/FRt2/detection_loss‚1.897714584585159 %loss/total_loss‚42.64951886271141$´»f[ûÕ 1loss/SRqqt4/assignment_loss‚1.539214524673298 .loss/FRt2/detection_loss‚1.897714584585159 %loss/total_loss‚42.64951886271141 2loss/SRqqt3/assignment_loss‚1.8613210434422773 /loss/SRqqt4/detection_loss‚1.91558374799788 .loss/FBt3/detection_loss‚2.891174920967647 epoch‚5 % _timestamp‚1.7884333403502083e+09 trainer/global_step‚6649 /loss/FRt2/assignment_loss‚3.245216030266977 /loss/FRt3/detection_loss‚3.4502933859825133 _runtime‚ 5025.05486093 0loss/FRt1/assignment_loss‚1.4644047247038947 0loss/FRt3/assignment_loss‚3.1450573444366454 0loss/FRt4/assignment_loss‚2.8955184671936967 /loss/FRt1/detection_loss‚1.5670505926012992 .loss/FRt4/detection_loss‚2.344895240737171 1loss/SRqqt1/detection_loss‚0.9885815089083698 .loss/FBt2/detection_loss‚2.337576389312744 /loss/FBt4/detection_loss‚2.5120398104190826 2loss/SRqqt2/assignment_loss‚1.6655458400398493 1loss/SRqqt2/detection_loss‚1.8615443593263625 0loss/SRqqt3/detection_loss‚2.347577988224871 2loss/SRqqt1/assignment_loss‚0.7647526080222571 1loss/FBt1/assignment_loss‚0.18376289788112102 1loss/FBt2/assignment_loss‚0.15154271781567433 0loss/FBt3/assignment_loss‚0.5795348157574024 /loss/FBt4/assignment_loss‚0.444189742517968 /loss/FBt1/detection_loss‚0.5954255768972472 _step‚546¢À¾~“ü‚(:ˆ ˧åÔ¨äÿŽ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.767' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.5534 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ \µ$ý‚(:™ ˧åÔêó”‘) disk./.usagePercent‚73.87535634200705 disk.nvme0n1.in‚ 579.4609375% disk./.usageGB‚1100.7961959838867 disk.sda.in‚0& disk.nvme0n1.out‚29579.29345703125 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 579.4609375 disk.rbd1.in‚0 network.sent‚ 1.5287682e+07 disk.sda3.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 196.640625! disk.nvme0c0n1.in‚ 579.4609375 disk.sda2.out‚0% proc.memory.availableMB‚ 948.09375 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda1.out‚0 cpu‚25.341224495016025 disk.sda3.out‚ 196.640625 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2872.4296875 proc.cpu.threads‚17 disk.rbd2.out‚ 146.5390625 disk.drbd1777.out‚0 disk.dm-1.out‚ 196.640625 disk.rbd1.out‚ 2.759765625( disk.nvme0c0n1.out‚29579.29345703125 disk.sda2.in‚0 network.recv‚ 1.808267e+06# disk.dm-0.out‚29579.29345703125$ memory_percent‚94.21329498291016* proc.memory.percent‚17.531919479370117yÌ~q’þ‚(:‡ Ú§åÔͺí gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹9ÄŒ"ÿ‚(:— Ú§åÔé — network.sent‚ 1.5382737e+07! disk.dm-0.out‚29580.580078125$ disk.nvme0n1.out‚29580.580078125 disk.rbd1.out‚ 2.759765625 disk.sda2.in‚0$ memory_percent‚94.21148300170898" proc.memory.rssMB‚ 2872.4296875 disk.dm-0.in‚ 579.4609375 disk.drbd1777.in‚0 disk.rbd3.in‚0& proc.memory.availableMB‚ 948.390625 network.recv‚ 1.811608e+06 disk.sda.out‚ 197.0703125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 579.4609375 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 197.0703125& disk.nvme0c0n1.out‚29580.580078125 disk.sda3.in‚0 cpu‚25.34032576704911 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd0.in‚0* proc.memory.percent‚17.531919479370117% disk./.usageGB‚1100.7961959838867 disk.sda3.out‚ 197.0703125 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 146.55078125) disk./.usagePercent‚73.87535634200705! disk.nvme0c0n1.in‚ 579.4609375Bìá22€- _wandbruntime‚5059  _runtime‚5059mfP'Éà _step‚547 ' lr-AdamW/pg1‚0.00011169203706518596 ' lr-AdamW/pg2‚0.00011169203706518596 trainer/global_step‚6699 % _timestamp‚1.7884333745011313e+09 _runtime‚5059.205913126²áÒaÎ‚È _step‚547 ' lr-AdamW/pg1‚0.00011169203706518596 ' lr-AdamW/pg2‚0.00011169203706518596 trainer/global_step‚6699 % _timestamp‚1.7884333745011313e+09 _runtime‚5059.205913126£6›xx2ƒ- _wandbruntime‚5059  _runtime‚5059”RæIá'„ÛO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚5059.206939209 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32012304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97318686 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 223713883 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5606946914816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56780062 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚394790660751360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97318688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5586960327186 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126395197 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚389183731839488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389183731839488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5607603277312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126395197 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88792807 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425011429773824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚378990909418496 trainer/global_step‚6699 % _timestamp‚1.7884333745018892e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5606946914816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430893547693056 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚378990892379136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚384577869745682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚394791335116800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88792364 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5607603277312 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5586305889430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97318686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56780060 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚389183713836544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚384577198268566 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126415235 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394790660751360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚389183713836544 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚389183713836544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚384577198268^üf!566 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126415235 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394790660751360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚389183713836544 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88792807 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425011429773824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚378990909418496 trainer/global_step‚6699 % _timestamp‚1.7884333745018892e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5606946914816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430893547693056 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚378990892379136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚384577869745682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚394791335116800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88792364 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5607603277312 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5586305889430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97318686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56780060 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚389240066875392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32016836 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚384633590311414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126413651 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126433715 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚379045796509696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223746353 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88805064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚394848507596800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚394847833218048 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97332702 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5608422718464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425073080511488 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223766419 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5587776762358 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223746353 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389240084878336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5607766342656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32017284 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430956056162304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56788230 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚394848507596800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5608422718464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚6699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389240066875392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5607766342656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97332702 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5587122324394 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884333751669762e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126433715 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5882951710720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425073104451584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56788228 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97332704 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379045813549056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430ü‡ã·ð&ˆêM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88805064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚394848507596800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚394847833218048 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97332702 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5608422718464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425073080511488 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223766419 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5587776762358 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223746353 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389240084878336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5607766342656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32017284 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430956056162304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56788230 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚394848507596800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5608422718464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚6699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389240066875392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5607766342656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97332702 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5587122324394 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884333751669762e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126433715 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5882951710720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425073104451584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56788228 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97332704 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379045813549056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚5059.872137822 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88805514 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5882790197248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚549 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223766419 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚389240066875392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32016836 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚384633590311414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126413651 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126433715 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚379045796509696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223746353 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 Bá'”ÛO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 % _timestamp‚1.7884334093365183e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98019486 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387363911613090 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5627795666594 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5924932297216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387363240135974 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚434018898954240 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127321249 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _runtime‚5094.041453345 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127341287 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98019488 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚397649972865024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5648589083136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428093966657024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98019488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89428158 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32239254 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428093990597120 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397649298499584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57188460 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32239696 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚434019084421120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397649972865024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5627141228838 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5647932720640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚381736115946496 _step‚552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5648589083136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57188462 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚6749 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127341287 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚381736098907136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392001365778944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392001383781888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89428158 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32239254 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428093990597120 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397649298499584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57188460 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32239696 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚434019084421120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397649972865024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5627141228838 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5647932720640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚381736115946496 _step‚552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5648589083136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57188462 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚6749 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127341287 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚381736098907136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392001365778944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392001383781888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 % _timestamp‚1.7884334093365183e+09¨æ2l2–- _wandbruntime‚5094  _runtime‚5094_ òrë&—åM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89440378 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127359767 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127339703 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5925792755712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚434081595603968 trainer/global_step‚6749 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381791020077056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392057718817792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98033502 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚397706470966272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387418960701498 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98033504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5649408524288 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127339703 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57196628 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392057736820736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57196630 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98033504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89440811 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5648752148480 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127359767 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5627957663802 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5649408524288 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428155641334784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225393271 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392057718817792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚381791003037696 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5628612101766 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225393271 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5925954269184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5648752148480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0B8êi˘êM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_byteßÙ%&s.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5628612101766 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225393271 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5925954269184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5648752148480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89440378 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127359767 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127339703 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5925792755712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚434081595603968 trainer/global_step‚6749 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381791020077056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392057718817792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98033502 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚397706470966272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387418960701498 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98033504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5649408524288 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127339703 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57196628 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392057736820736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57196630 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98033504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89440811 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5648752148480 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127359767 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5627957663802 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5649408524288 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428155641334784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225393271 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392057718817792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚381791003037696 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚400507936247808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437144772448768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884334444872909e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394819035724288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚400508610613248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90063560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚394819035724288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚400508610613248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5689574888960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437144591176192 trainer/global_step‚6799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57596860 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚431176527480320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944óÇç5%¥ßO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226967587 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚400507936247808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437144772448768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884334444872909e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394819035724288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚400508610613248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90063560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚394819035724288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚400508610613248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5689574888960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437144591176192 trainer/global_step‚6799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57596860 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚431176527480320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576¬y¢‘}2¦- _wandbruntime‚5129  _runtime‚5129mlô.ë&§åM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390205674046230 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5689737954304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98734304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394875370760192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5968949945856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400565783093248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431238154277888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚5129.852729761 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227000057 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98734304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128265755 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437207104223744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227020123 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227020123 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98734302 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5969111459328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384536209565696 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400565783093248 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57605028 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128285819 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚431238178217984 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884334451473145e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5690394330112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128265755 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90075835 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32471242 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384536226605056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437207289677312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _step‚557 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128ïV ð&¨êM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128285819 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚431238178217984 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884334451473145e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5690394330112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128265755 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90075835 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32471242 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384536226605056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437207289677312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _step‚557 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390205674046230 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5689737954304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98734304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394875370760192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5968949945856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400565783093248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431238154277888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚5129.852729761 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227000057 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98734304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128265755 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437207104223744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227020123 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227020123 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98734302 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5969111459328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384536209565696 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400565783093248 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57605028 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0­œÆ2©- _wandbruntime‚5129  _runtime‚5129´ÏÂ9\ªÖ /loss/FRt1/assignment_loss‚1.018832029180324 0loss/FRt4/assignment_loss‚3.1284752304737387 2loss/SRqqt1/assignment_loss‚0.7723873508961073 /loss/FRt4/detection_loss‚2.4988191620661664 $ _timestamp‚1.788433445149158e+09 0loss/FRt3/assignment_loss‚2.8012967721009865 1loss/FBt2/assignment_loss‚0.14713774874107913 /loss/FRt1/detection_loss‚1.8253522684916537 /loss/FBt1/detection_loss‚0.5756354726107291 1loss/SRqqt3/detection_loss‚2.5676266262025544 %loss/total_loss‚41.28666843633091 _runtime‚5129.853788727 1loss/SRqqt2/assignment_loss‚1.058657682416114 1loss/SRqqt2/detection_loss‚1.8955908200957559 _step‚558 /loss/FRt2/assignment_loss‚3.255210598309835 2loss/SRqqt3/assignment_loss‚1.7690667514548157 0loss/FBt4/assignment_loss‚0.9560760503324369 .loss/FRt3/detection_loss‚2.677673128935007 1loss/SRqqt4/detection_loss‚1.8686471494535606 trainer/global_step‚6799 1loss/FBt3/assignment_loss‚0.17179096219063336 1loss/SRqqt1/detection_loss‚0.8462874815400158 /loss/FBt4/detection_loss‚2.6278336892525354 epoch‚5 2loss/SRqqt4/assignment_loss‚1.3025747724071455 1loss/FBt1/assignment_loss‚0.20153376774536505 .loss/FRt2/detection_loss‚2.442764217654864 /loss/FBt2/detection_loss‚2.8914851546287537 /loss/FBt3/detection_loss‚1.9859135491507394a«Û /loss/FRt1/assignment_loss‚1.018832029180324 0loss/FRt4/assignment_loss‚3.1284752304737387 2loss/SRqqt1/assignment_loss‚0.7723873508961073 /loss/FRt4/detection_loss‚2.4988191620661664 $ _timestamp‚1.788433445149158e+09 0loss/FRt3/assignment_loss‚2.8012967721009865 1loss/FBt2/assignment_loss‚0.14713774874107913 /loss/FRt1/detection_loss‚1.8253522684916537 /loss/FBt1/detection_loss‚0.5756354726107291 1loss/SRqqt3/detection_loss‚2.5676266262025544 %loss/total_loss‚41.28666843633091 _runtime‚5129.853788727 1loss/SRqqt2/assignment_loss‚1.058657682416114 1loss/SRqqt2/detection_loss‚1.8955908200957559 _step‚558 /loss/FRt2/assignment_loss‚3.255210598309835 2loss/SRqqt3/assignment_loss‚1.7690667514548157 0loss/FBt4/assignment_loss‚0.9560760503324369 .loss/FRt3/detection_loss‚2.677673128935007 1loss/SRqqt4/detection_loss‚1.8686471494535606 trainer/global_step‚6799 1loss/FBt3/assignment_loss‚0.17179096219063336 1loss/SRqqt1/detection_loss‚0.8462874815400158 /loss/FBt4/detection_loss‚2.6278336892525354 epoch‚5 2loss/SRqqt4/assignment_loss‚1.3025747724071455 1loss/FBt1/assignment_loss‚0.20153376774536505 .loss/FRt2/detection_loss‚2.442764217654864 /loss/FBt2/detection_loss‚2.8914851546287537 /loss/FBt3/detection_loss‚1.9859135491507394®É…… “¬‚(:ˆ ¥¨åÔ£÷€Ž gpu.0.gpu‚46.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.079' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4158 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nï¹.­‚(:£ ¥¨åÔðá°‘ network.sent‚ 1.5612566e+07 disk.dm-1.out‚ 200.24609375 disk.rbd1.out‚ 2.759765625 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚29981.49072265625 disk.sda1.out‚0& disk.nvme0n1.out‚29981.49072265625 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 579.4609375 disk.rbd0.out‚0" memory_percent‚94.215989112854) disk./.usagePercent‚73.87556652419266( proc.memory.availableMB‚ 947.65234375% disk./.usageGB‚1100.7993278503418 disk.sda2.in‚0 disk.sda3.out‚ 200.24609375 disk.dm-0.in‚ 579.4609375" proc.memory.rssMB‚ 2872.4296875* proc.memory.percent‚17.531919479370117 cpu‚25.33604196398895 disk.rbd1.in‚0( disk.nvme0c0n1.out‚29981.49072265625! disk.rbd2.out‚148.36767578125 disk.sda.out‚ 200.24609375 proc.cpu.threads‚17 network.recv‚ 1.832567e+06! disk.nvme0c0n1.in‚ 579.4609375 disk.sda2.out‚0 disk.sda.in‚0÷ ‚µ®‚(:’ ´¨åÔà—‚Ž gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.59' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.317999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—£tN,¯‚(:¡ ´¨åÔ‚´¡ disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 2.759765625 disk.rbd0.out‚0 disk.sda1.in‚0! disk.rbd2.out‚149.89501953125 cpu‚25.33522352441818 disk.sda2.out‚0 network.recv‚ 1.838385e+06 disk.dm-0.in‚ 579.4609375% disk.nvme0n1.out‚30081.0419921875' disk.nvme0c0n1.out‚30081.0419921875 disk.sda2.in‚0# memory_percent‚94.2145586013794) disk./.usagePercent‚73.87556652419266% disk./.usageGB‚1100.7993278503418 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 579.4609375 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.5720636e+07 disk.rbd3.in‚0 disk.sda.out‚ 200.82421875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0" disk.dm-0.out‚30081.0419921875 disk.dm-1.out‚ 200.82421875 disk.sda3.out‚ 200.82421875" proc.memory.rssMB‚ 2872.4296875! disk.nvme0c0n1.in‚ 579.4609375 disk.sda3.in‚0( proc.memory.availableMB‚ 947.88671875* proc.memory.percent‚17.5319194793701171~už°‚(:“ èåÔé¤× gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.471' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.294199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿:Ô&±‚(:› èåÔ»ñé disk.sda2.out‚0" disk.dm-0.out‚30180.1474609375 disk.rbd3.out‚0 disk.dm-1.out‚ 201.453125" proc.memory.rssMB‚ 2872.4296875) disk./.usagePercent‚73.87557446042624( proc.memory.availableMB‚ 947.89453125 cpu‚25.33439293350018 proc.cpu.threads‚17 network.recv‚ 1.842029e+06 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 579.4609375 disk.rbd1.out‚ 2.759765625$ memory_percent‚94.21451091766357* proc.memory.percent‚17.531919479370117 disk.rbd0.out‚0 disk.sda3.out‚ 201.453125 disk.rbd3.in‚0 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚150.02001953125 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 579.4609375 disk.sda3.in‚0 disk.sda.out‚ 201.453125 disk.sda1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚30180.1474609375 network.sent‚ 1.5725977e+07$ disk./.usageGB‚1100.799446105957! disk.nvme0c0n1.in‚ 579.4609375' disk.nvme0c0n1.out‚30180.1474609375$s¨ 2²- _wandbruntime‚5164  _runtime‚51645Î7Gɳà trainer/global_step‚6849 % _timestamp‚1.7884334795844276e+09 _runtime‚5164.289176354 _step‚559 ' lr-AdamW/pg1‚0.00010678825275991335 ' lr-AdamW/pg2‚0.00010678825275991335HAyÎ´È trainer/global_step‚6849 % _timestamp‚1.7884334795844276e+09 _runtime‚5164.289176354 _step‚559 ' lr-AdamW/pg1‚0.00010678825275991335 ' lr-AdamW/pg2‚0.00010678825275991335¯ß²Â2µ- _wandbruntime‚5164  _runtime‚5164;+}6}¶ÛO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434259040423424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403367248361472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434259064363520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129193391 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397636687666688 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99421086 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32694754 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99421086 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129173353 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚397636687666688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5729904332288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440270135604736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚387226529002496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397636669663744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228614479 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129173353 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5730560694784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403367248361472 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6011265096704 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392935995347906 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397636669663744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403366573996032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5730560694784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440270329460224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99421088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32694302 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5708811907654 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5709466345410 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚6849 % _timestamp‚1.7884334795851774e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403366573996032 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129193391 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90699562iÁ?Jæ'·àO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403367248361472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434259064363520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129193391 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397636687666688 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99421086 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32694754 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99421086 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129173353 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚397636687666688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5729904332288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440270135604736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚387226529002496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397636669663744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228614479 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129173353 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5730560694784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403367248361472 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6011265096704 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392935995347906 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397636669663744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403366573996032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5730560694784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440270329460224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99421088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32694302 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5708811907654 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5709466345410 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚6849 % _timestamp‚1.7884334795851774e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403366573996032 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129193391 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90699562 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434259040423424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚560°]FŽo2¸- _wandbruntime‚5164  _runtime‚5164P5}ùë&¹åM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99435102 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚392991715913638 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884334802474008e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397693022702592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6012128110080 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403424420841472 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129211871 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90712172 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228646975 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99435104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚397693040705536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚397693040705536 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5710282780582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58013428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440332843211264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403424420841472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228646975 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚387281416093696 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228626909 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32698744 trainer/global_step‚6849 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129191807 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚387281433133056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6011974985216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0‚¢Ú®v ºêM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228646975 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99435104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚397693040705536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚397693040705536 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5710282780582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58013428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440332843211264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403424420841472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228646975 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚387281416093696 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228626909 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32698744 trainer/global_step‚6849 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129191807 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚387281433133056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6011974985216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99435102 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚392991715913638 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884334802474008e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397693022702592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6012128110080 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403424420841472 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129211871 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90712172 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0±)1%2»- _wandbruntime‚5164  _runtime‚5164ée Z¼Ô 1loss/FBt3/assignment_loss‚0.18732392968377098 /loss/FRt1/detection_loss‚1.7044691987969773 /loss/FBt1/detection_loss‚0.7006772447733126 /loss/FRt2/assignment_loss‚2.582094121920435 2loss/SRqqt3/assignment_loss‚1.7773335733347468 /loss/FBt4/assignment_loss‚0.136289038695395 0loss/FRt1/assignment_loss‚1.3710494916489784 /loss/FRt4/assignment_loss‚3.333385755849439 2loss/SRqqt2/assignment_loss‚1.4672015180641955 /loss/FRt4/detection_loss‚2.3884988884593166 /loss/FBt2/detection_loss‚2.9206595420837402 .loss/FBt3/detection_loss‚2.805724710226059 2loss/SRqqt4/assignment_loss‚1.7026525732989495 1loss/SRqqt3/detection_loss‚2.4111500283082328 epoch‚5 trainer/global_step‚6849 .loss/FBt4/detection_loss‚3.302634429931641 0loss/FRt3/assignment_loss‚3.3102921824301443 1loss/FBt2/assignment_loss‚0.05698053696930098 /loss/SRqqt2/detection_loss‚1.99839438599619 .loss/FRt2/detection_loss‚2.525865852832794 %loss/total_loss‚43.77343519998285 % _timestamp‚1.7884334802490568e+09 2loss/SRqqt1/assignment_loss‚0.9664887237611791 1loss/FBt1/assignment_loss‚0.17078472947887072 /loss/FRt3/detection_loss‚2.9650353058691947 1loss/SRqqt1/detection_loss‚1.0372670609766328 1loss/SRqqt4/detection_loss‚1.9511823765933514 _runtime‚5164.953706046 _step‚562ò{¹I_½Ù 0loss/FRt3/assignment_loss‚3.3102921824301443 1loss/FBt2/assignment_loss‚0.05698053696930098 /loss/SRqqt2/detection_loss‚1.99839438599619 .loss/FRt2/detection_loss‚2.525865852832794 %loss/total_loss‚43.77343519998285 % _timestamp‚1.7884334802490568e+09 2loss/SRqqt1/assignment_loss‚0.9664887237611791 1loss/FBt1/assignment_loss‚0.17078472947887072 /loss/FRt3/detection_loss‚2.9650353058691947 1loss/SRqqt1/detection_loss‚1.0372670609766328 1loss/SRqqt4/detection_loss‚1.9511823765933514 _runtime‚5164.953706046 _step‚562 1loss/FBt3/assignment_loss‚0.18732392968377098 /loss/FRt1/detection_loss‚1.7044691987969773 /loss/FBt1/detection_loss‚0.7006772447733126 /loss/FRt2/assignment_loss‚2.582094121920435 2loss/SRqqt3/assignment_loss‚1.7773335733347468 /loss/FBt4/assignment_loss‚0.136289038695395 0loss/FRt1/assignment_loss‚1.3710494916489784 /loss/FRt4/assignment_loss‚3.333385755849439 2loss/SRqqt2/assignment_loss‚1.4672015180641955 /loss/FRt4/detection_loss‚2.3884988884593166 /loss/FBt2/detection_loss‚2.9206595420837402 .loss/FBt3/detection_loss‚2.805724710226059 2loss/SRqqt4/assignment_loss‚1.7026525732989495 1loss/SRqqt3/detection_loss‚2.4111500283082328 epoch‚5 trainer/global_step‚6849 .loss/FBt4/detection_loss‚3.302634429931641²Ÿ`L¯‘¾‚(:† Ò¨åÔû£ë gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.5844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç OC,¿‚(:¡ Ò¨åÔÁ„ó disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0% proc.memory.availableMB‚ 948.15625& disk.nvme0n1.out‚30279.52001953125 disk.rbd2.in‚0) disk./.usagePercent‚73.87558137262967% disk./.usageGB‚1100.7995491027832 disk.dm-1.out‚ 202.08203125! disk.rbd2.out‚150.03564453125 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 579.4609375 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 202.08203125 disk.rbd3.out‚0# disk.dm-0.out‚30279.52001953125 network.sent‚ 1.5730267e+07 disk.rbd0.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 579.4609375 network.recv‚ 1.844957e+06 disk.sda1.in‚0( disk.nvme0c0n1.out‚30279.52001953125 disk.sda3.out‚ 202.08203125 disk.drbd1777.in‚0# memory_percent‚94.2129135131836* proc.memory.percent‚17.531919479370117 disk.rbd1.out‚ 2.759765625! disk.nvme0c0n1.in‚ 579.4609375" proc.memory.rssMB‚ 2872.4296875 cpu‚25.33352394745982ä.Db“À‚(:ˆ á¨åÔ‹†Ø gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.264' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.6528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñç4,Á‚(:¡ á¨åԃͳ‘ disk.drbd1777.in‚0# disk.dm-0.out‚30378.60205078125 disk.sda1.in‚0 disk.dm-1.out‚ 202.69140625 disk.rbd0.out‚0 network.sent‚ 1.583809e+07 disk.sda1.out‚0 disk.rbd3.in‚0* proc.memory.percent‚17.531919479370117 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 579.4609375 disk.rbd2.out‚150.1845703125$ memory_percent‚94.21682357788086 proc.cpu.threads‚17 disk.sda.out‚ 202.69140625 disk.sda3.in‚0 disk.sda3.out‚ 202.69140625 disk.drbd1777.out‚0& proc.memory.availableMB‚ 947.515625$ disk./.usageGB‚1100.799472808838 disk.dm-0.in‚ 579.4609375 disk.dm-1.in‚0& disk.nvme0n1.out‚30378.60205078125 disk.rbd1.out‚ 2.759765625( disk.nvme0c0n1.out‚30378.60205078125 disk.rbd2.in‚0" proc.memory.rssMB‚ 2872.4296875 network.recv‚ 1.850016e+06 disk.nvme0n1.in‚ 579.4609375 cpu‚25.332703436864175 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚73.87557625247896 disk.sda.in‚0$ƒu2Â- _wandbruntime‚5199  _runtime‚5199ä¿÷®ÉÃà _step‚563 ' lr-AdamW/pg1‚0.00010515592280576655 ' lr-AdamW/pg2‚0.00010515592280576655 trainer/global_step‚6899 % _timestamp‚1.7884335146923552e+09 _runtime‚5199.3970831794¦—ÎÄÈ % _timestamp‚1.7884335146923552e+09 _runtime‚5199.397083179 _step‚563 ' lr-AdamW/pg1‚0.00010515592280576655 ' lr-AdamW/pg2‚0.00010515592280576655 trainer/global_step‚6899³ßBro2Å- _wandbruntime‚5199  _runtime‚5199ºD–ä'ÆÞO UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100121888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130119443 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437341577306624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5749647247062 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130099405 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚406225886109696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400454321606144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406225886109696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400454339609088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054457541632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437341601246720 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395722037215314 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58413662 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚389971718491136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5771546500608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400454321606144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100121886 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91335382 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100121886 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32921259 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788433514693052e+09 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚406225211744256 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6054287626240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚6899 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5771546500608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32921720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395721365738198 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5770890138112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100121888 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389971735530496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91334919 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406225211744256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58413660 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¸= .Þ%ÇãO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400454321606144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100121886 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91335382 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100121886 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32921259 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788433514693052e+09 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚406225211744256 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6054287626240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚6899 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5771546500608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32921720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395721365738198 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5770890138112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100121888 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389971735530496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91334919 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406225211744256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58413660 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100121888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130119443 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437341577306624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5749647247062 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130099405 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚406225886109696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400454321606144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406225886109696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400454339609088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054457541632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437341601246720 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395722037215314 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58413662 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚389971718491136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.s©Ê,ø mall_pool.allocated‚ 5771546500608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414´µÁŠÀ2È- _wandbruntime‚5200  _runtime‚5200eÝÚmï&ÉéM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5750463682026 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100135902 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚406282384210944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230253761 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100135904 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91347606 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32926235 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230273827 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100135904 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5771709565952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230273827 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5772365941760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚406283058589696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91348065 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚5200.057423639 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5772365941760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚565 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130117859 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6055150637568 trainer/global_step‚6899 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5771709565952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437403251984384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚395777086303722 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6055312151040 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437403228044288 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100135902 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130117859 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚443458564135424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58421830 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58421828 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844ùìœô&ÊîM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100135902 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130117859 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚443458564135424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58421830 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58421828 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5750463682026 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100135902 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚406282384210944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230253761 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100135904 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91347606 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32926235 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230273827 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100135904 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5771709565952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230273827 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5772365941760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚406283058589696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91348065 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚5200.057423639 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5772365941760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚565 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130117859 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6055150637568 trainer/global_step‚6899 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5771709565952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437403251984384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚395777086303722 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6055312151040 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437403228044288 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521µÁ¶Š2Ë- _wandbruntime‚5200  _runtime‚5200øû YÌÓ 0loss/FRt3/assignment_loss‚3.6257165173689523 1loss/SRqqt2/detection_loss‚1.9365431497926298 -loss/FBt3/detection_loss‚2.64522651831309 2loss/SRqqt1/assignment_loss‚1.1857392731738292 1loss/FBt1/assignment_loss‚0.11890397648266889 2loss/SRqqt4/assignment_loss‚1.4402202469296754 /loss/FBt1/detection_loss‚0.5516618435060949 _step‚566 trainer/global_step‚6899 2loss/SRqqt2/assignment_loss‚1.4733029576423375 2loss/SRqqt3/assignment_loss‚1.9293549799493381 /loss/FRt1/detection_loss‚1.8460905080322514 %loss/total_loss‚44.50031892414502 /loss/FRt1/assignment_loss‚1.586994161660021 1loss/FBt2/assignment_loss‚0.17636926698032768 1loss/FBt4/assignment_loss‚0.37670433001879317 /loss/FRt2/detection_loss‚2.7052497181453203 .loss/FRt4/detection_loss‚2.266323787986111 /loss/SRqqt1/detection_loss‚1.34811170891685 .loss/FBt2/detection_loss‚2.920829176902771 % _timestamp‚1.7884335153538787e+09 0loss/SRqqt3/detection_loss‚2.026444854055132 1loss/SRqqt4/detection_loss‚1.8188254271944364 .loss/FRt3/detection_loss‚2.759673269258605 .loss/FBt4/detection_loss‚2.542995246557089 0loss/FRt2/assignment_loss‚3.3742940896435787 0loss/FRt4/assignment_loss‚3.4077701040034025 1loss/FBt3/assignment_loss‚0.43697381163171184 epoch‚5 _runtime‚5200.058493265Cn¦^ÍØ 0loss/FRt2/assignment_loss‚3.3742940896435787 0loss/FRt4/assignment_loss‚3.4077701040034025 1loss/FBt3/assignment_loss‚0.43697381163171184 epoch‚5 _runtime‚5200.058493265 0loss/FRt3/assignment_loss‚3.6257165173689523 1loss/SRqqt2/detection_loss‚1.9365431497926298 -loss/FBt3/detection_loss‚2.64522651831309 2loss/SRqqt1/assignment_loss‚1.1857392731738292 1loss/FBt1/assignment_loss‚0.11890397648266889 2loss/SRqqt4/assignment_loss‚1.4402202469296754 /loss/FBt1/detection_loss‚0.5516618435060949 _step‚566 trainer/global_step‚6899 2loss/SRqqt2/assignment_loss‚1.4733029576423375 2loss/SRqqt3/assignment_loss‚1.9293549799493381 /loss/FRt1/detection_loss‚1.8460905080322514 %loss/total_loss‚44.50031892414502 /loss/FRt1/assignment_loss‚1.586994161660021 1loss/FBt2/assignment_loss‚0.17636926698032768 1loss/FBt4/assignment_loss‚0.37670433001879317 /loss/FRt2/detection_loss‚2.7052497181453203 .loss/FRt4/detection_loss‚2.266323787986111 /loss/SRqqt1/detection_loss‚1.34811170891685 .loss/FBt2/detection_loss‚2.920829176902771 % _timestamp‚1.7884335153538787e+09 0loss/SRqqt3/detection_loss‚2.026444854055132 1loss/SRqqt4/detection_loss‚1.8188254271944364 .loss/FRt3/detection_loss‚2.759673269258605 .loss/FBt4/detection_loss‚2.542995246557089¶G€“΂(:ˆ ð¨åÔ”›Ô gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.298' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4596 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`ä#(Ï‚(: ð¨åÔ„„˜) disk./.usagePercent‚73.87557625247896 disk.rbd1.out‚ 2.759765625 disk.rbd2.in‚0' proc.memory.availableMB‚ 947.9921875" proc.memory.rssMB‚ 2872.4296875* proc.memory.percent‚17.531919479370117$ disk./.usageGB‚1100.799472808838 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda.out‚ 203.47265625 disk.sda1.out‚0 disk.rbd3.out‚0! disk.dm-0.out‚30379.177734375 disk.dm-1.out‚ 203.47265625 disk.nvme0n1.in‚ 579.4609375 disk.rbd1.in‚0 disk.sda2.out‚0 network.sent‚ 1.5840984e+07 disk.sda1.in‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚30379.177734375 disk.sda3.out‚ 203.47265625 proc.cpu.threads‚17 disk.rbd2.out‚150.2001953125 network.recv‚ 1.851889e+06 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.in‚0! disk.nvme0c0n1.in‚ 579.4609375$ disk.nvme0n1.out‚30379.177734375 cpu‚25.331952743383564 disk.dm-0.in‚ 579.4609375 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚94.21391487121582 disk.sda3.in‚0Q ³çЂ(:’ ÿ¨åÔÆ«ø gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.44' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.287999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 t+Æ#Ñ‚(:˜ ÿ¨åÔÁ„ª disk.rbd1.in‚0 disk.rbd0.in‚0" proc.memory.availableMB‚947.75* proc.memory.percent‚17.531919479370117 cpu‚25.331145111063208 network.recv‚ 1.858073e+06 disk.dm-0.in‚ 579.4609375 disk.sda.in‚0! disk.nvme0c0n1.in‚ 579.4609375' disk.nvme0c0n1.out‚30478.3173828125$ memory_percent‚94.21539306640625% disk./.usageGB‚1100.7995262145996 disk.sda3.out‚ 204.0859375 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 579.4609375 disk.sda3.in‚0" disk.dm-0.out‚30478.3173828125 disk.rbd0.out‚0 network.sent‚ 1.594988e+07 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚73.87557983658445 disk.sda.out‚ 204.0859375 disk.sda1.out‚0 disk.rbd2.out‚150.2626953125" proc.memory.rssMB‚ 2872.4296875 disk.rbd3.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚30478.3173828125 disk.rbd1.out‚ 2.759765625 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.dm-1.out‚ 204.0859375ÑÖ %2Ò- _wandbruntime‚5234  _runtime‚5234¶™õ?ÇÓÁ ' lr-AdamW/pg1‚0.00010352529244154301 ' lr-AdamW/pg2‚0.00010352529244154301 trainer/global_step‚6949 # _timestamp‚1.78843354950447e+09 _runtime‚5234.209224162 _step‚567„æÆ\ÌÔÆ ' lr-AdamW/pg1‚0.00010352529244154301 ' lr-AdamW/pg2‚0.00010352529244154301 trainer/global_step‚6949 # _timestamp‚1.78843354950447e+09 _runtime‚5234.209224162 _step‚567·*Ê?2Õ- _wandbruntime‚5234  _runtime‚5234 f‡ÖÞO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚409084523857920 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91971555 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6949 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6097676297728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884335495052004e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100822688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚446521814427648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392716925019136 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100822686 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131045495 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58822062 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231848143 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚392716942058496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚440424138129920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131045495 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚409084523857920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚403271991551488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5812532306432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚403271973548544 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚409083849492480 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100822686 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91972007 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58822060 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5811875943936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _step‚568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5791137024226 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚398508079082722 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚ 5234.21013602 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33149495 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚403271991551488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚440424114189824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398507407605606 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131025457 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409083849492480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6097514770944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100822688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231868183 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5811875943936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5790482586470 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928Îh.Æé'×ãO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚403271991551488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚440424114189824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398507407605606 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131025457 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409083849492480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6097514770944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100822688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231868183 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5811875943936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5790482586470 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚409084523857920 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91971555 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6949 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6097676297728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884335495052004e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100822688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚446521814427648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392716925019136 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100822686 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131045495 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58822062 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231848143 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚392716942058496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚440424138129920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131045495 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚409084523857920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚403271991551488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5812532306432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚403271973548544 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚409083849492480 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100822686 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91972007 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58822060 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5811875943936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _step‚568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5791137024226 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚398508079082722 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚ 5234.21013602 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33149495 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844¸¨ã†A2Ø- _wandbruntime‚5234  _runtime‚5234šÝÏ£ï&ÙéM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131043911 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100836702 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚446584128230400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6098363302912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91984727 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231900679 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚440485764927488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100836704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33154497 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5791299021434 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5812695371776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚446584322072576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚409141021959168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884335501608248e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5812695371776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚409141696337920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33154042 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚440485788867584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398563128171130 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392771846189056 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131063975 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5813351747584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5791953459398 _runtime‚5234.865981648 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100836704 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100836702 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6098533204992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6949 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231900679 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231880613 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚409141696337920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚403328326587392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131063975 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91984270 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚403328326587392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58830230 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392771829149696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¢þÝ ÚîM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231880613 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚409141696337920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚403328326587392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131063975 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91984270 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚403328326587392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58830230 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392771829149696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131043911 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100836702 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚446584128230400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6098363302912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91984727 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231900679 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚440485764927488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100836704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33154497 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5791299021434 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5812695371776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚446584322072576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚409141021959168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884335501608248e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5812695371776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚409141696337920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_s&ù\ðplit.small_pool.freed‚33154042 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚440485788867584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398563128171130 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392771846189056 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131063975 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5813351747584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5791953459398 _runtime‚5234.865981648 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100836704 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100836702 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6098533204992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6949 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231900679 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430¹Ü” 2Û- _wandbruntime‚5234  _runtime‚5234£·´VÜÐ /loss/FRt3/detection_loss‚2.6899317037314177 .loss/FBt2/detection_loss‚2.307097691756028 1loss/FBt1/assignment_loss‚0.26678683137141473 1loss/SRqqt2/detection_loss‚1.9472811659988092 $ _timestamp‚1.788433550162543e+09 _step‚570 /loss/FBt3/detection_loss‚2.6870387537138805 epoch‚5 &loss/total_loss‚43.957995718048096 /loss/FRt4/assignment_loss‚2.763944968234661 2loss/SRqqt2/assignment_loss‚1.4661980204056113 /loss/FRt2/detection_loss‚2.5720672425700397 /loss/FRt1/assignment_loss‚1.598265853463387 0loss/FBt2/assignment_loss‚0.3755320159700484 1loss/FBt3/assignment_loss‚0.16256605409684458 1loss/SRqqt1/detection_loss‚0.9342078013873812 _runtime‚5234.867169105 /loss/FRt2/assignment_loss‚2.754537218954505 1loss/FBt4/assignment_loss‚0.20072151198983193 /loss/FRt4/detection_loss‚2.1460408672343854 /loss/FBt1/detection_loss‚0.8097227823455118 .loss/FBt4/detection_loss‚4.003204500675201 trainer/global_step‚6949 0loss/SRqqt1/assignment_loss‚0.99810313155402 2loss/SRqqt3/assignment_loss‚1.6043944091846545 -loss/FRt1/detection_loss‚1.86166784638653 0loss/SRqqt3/detection_loss‚2.670653315881888 0loss/SRqqt4/detection_loss‚1.661396904124154 /loss/FRt3/assignment_loss‚4.022066663950682 2loss/SRqqt4/assignment_loss‚1.45456846306721380æë[ÝÕ /loss/FRt1/assignment_loss‚1.598265853463387 0loss/FBt2/assignment_loss‚0.3755320159700484 1loss/FBt3/assignment_loss‚0.16256605409684458 1loss/SRqqt1/detection_loss‚0.9342078013873812 _runtime‚5234.867169105 /loss/FRt2/assignment_loss‚2.754537218954505 1loss/FBt4/assignment_loss‚0.20072151198983193 /loss/FRt4/detection_loss‚2.1460408672343854 /loss/FBt1/detection_loss‚0.8097227823455118 .loss/FBt4/detection_loss‚4.003204500675201 trainer/global_step‚6949 0loss/SRqqt1/assignment_loss‚0.99810313155402 2loss/SRqqt3/assignment_loss‚1.6043944091846545 -loss/FRt1/detection_loss‚1.86166784638653 0loss/SRqqt3/detection_loss‚2.670653315881888 0loss/SRqqt4/detection_loss‚1.661396904124154 /loss/FRt3/assignment_loss‚4.022066663950682 2loss/SRqqt4/assignment_loss‚1.4545684630672138 /loss/FRt3/detection_loss‚2.6899317037314177 .loss/FBt2/detection_loss‚2.307097691756028 1loss/FBt1/assignment_loss‚0.26678683137141473 1loss/SRqqt2/detection_loss‚1.9472811659988092 $ _timestamp‚1.788433550162543e+09 _step‚570 /loss/FBt3/detection_loss‚2.6870387537138805 epoch‚5 &loss/total_loss‚43.957995718048096 /loss/FRt4/assignment_loss‚2.763944968234661 2loss/SRqqt2/assignment_loss‚1.4661980204056113 /loss/FRt2/detection_loss‚2.5720672425700397ºGiî’Þ‚(:‡ Ž©åÔò£ô gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚98.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.7524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h³ˆ(ß‚(: Ž©åÔìߊ network.sent‚ 1.5955114e+07 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚0$ disk.nvme0n1.out‚30577.689453125 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 579.4609375 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 2.759765625$ disk./.usageGB‚1100.798698425293& disk.nvme0c0n1.out‚30577.689453125 disk.rbd2.out‚151.7744140625 disk.sda1.out‚0 disk.dm-1.out‚ 204.58203125 disk.nvme0n1.in‚ 579.4609375$ memory_percent‚94.21553611755371 network.recv‚ 1.862016e+06) disk./.usagePercent‚73.87552428294953 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 204.58203125' proc.memory.availableMB‚ 947.7265625 cpu‚25.330298483414932 disk.sda3.in‚0 disk.sda3.out‚ 204.58203125 disk.dm-0.in‚ 579.4609375" proc.memory.rssMB‚ 2872.4296875* proc.memory.percent‚17.531919479370117 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0! disk.dm-0.out‚30577.689453125 disk.rbd1.in‚0iyøžà‚(:“ ©åÔçôÎ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.661' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.132199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(ýq)á‚(:ž ©åÔͪÍ disk.sda1.out‚0 network.recv‚ 1.866064e+06% disk./.usageGB‚1100.7988204956055( disk.nvme0c0n1.out‚30676.85400390625 disk.rbd0.in‚0$ memory_percent‚94.21594142913818 cpu‚25.32949652525302 network.sent‚ 1.6061861e+07 disk.dm-1.out‚ 205.078125 disk.nvme0n1.in‚ 579.4609375 disk.rbd1.out‚ 2.759765625 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 947.66015625 disk.sda2.in‚0 disk.sda3.out‚ 205.078125) disk./.usagePercent‚73.87553247519062 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 579.4609375 disk.rbd0.out‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.531919479370117 proc.cpu.threads‚17& disk.nvme0n1.out‚30676.85400390625 disk.dm-0.in‚ 579.4609375# disk.dm-0.out‚30676.85400390625 disk.sda.out‚ 205.078125 disk.sda1.in‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2872.4296875 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd2.out‚151.9150390625 disk.drbd1777.in‚0‚97ª“â‚(:ˆ ¬©åÔʲû gpu.0.gpu‚33.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚140.547' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.1094 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"a¤0ã‚(:¥ ¬©åÔþº¦ disk.sda2.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 601.26953125 disk.nvme0n1.in‚ 601.26953125 disk.rbd2.in‚0 disk.sda3.out‚ 205.6953125# disk.dm-0.out‚30677.50537109375 disk.sda.out‚ 205.6953125 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 601.26953125 cpu‚25.328659756264962 disk.rbd2.out‚151.9150390625 disk.sda1.in‚0( disk.nvme0c0n1.out‚30677.50537109375 proc.cpu.threads‚17 disk.dm-1.out‚ 205.6953125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 947.66796875 disk.dm-1.in‚0 network.sent‚ 1.6066653e+07) disk./.usagePercent‚73.87560287726254% disk./.usageGB‚1100.7998695373535" proc.memory.rssMB‚ 2872.4296875* proc.memory.percent‚17.531919479370117 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.759765625 disk.rbd0.out‚0 disk.sda3.in‚0 disk.drbd1777.out‚0$ memory_percent‚94.21589374542236& disk.nvme0n1.out‚30677.50537109375 network.recv‚ 1.869742e+06S—ú2ä- _wandbruntime‚5269  _runtime‚5269'9TÉåà ' lr-AdamW/pg1‚0.00010189670148816661 ' lr-AdamW/pg2‚0.00010189670148816661 trainer/global_step‚6999 % _timestamp‚1.7884335844591835e+09 _runtime‚5269.163967902 _step‚571mƒûÎæÈ _runtime‚5269.163967902 _step‚571 ' lr-AdamW/pg1‚0.00010189670148816661 ' lr-AdamW/pg2‚0.00010189670148816661 trainer/global_step‚6999 % _timestamp‚1.7884335844591835e+09»û$°2ç- _wandbruntime‚5269  _runtime‚5269ãúôä'èÞO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406089625490944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406089625490944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233474995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411942487240704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 $ _timestamp‚1.788433584459953e+09 _step‚572 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131971547 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92607742 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449647679341056 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101523486 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33377282 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101523488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131971547 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5852861749760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401294120950130 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401293449473014 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚411942487240704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449647493874176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5831972363634 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5831317925878 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443506651073024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131951509 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406089643493888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚406089643493888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6141004327936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401294120950130 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401293449473014 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚411942487240704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449647493874176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5831972363634 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5831317925878 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443506651073024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131951509 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406089643493888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚406089643493888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6141004327936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406089625490944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406089625490944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233474995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411942487240704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 $ _timestamp‚1.788433584459953e+09 _step‚572 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131971547 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92607742 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449647679341056 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101523486 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889¼yÐBÎ2ê- _wandbruntime‚5269  _runtime‚5269åt³q ëéM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233507465 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101537504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚406145978529792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233527531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131969963 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚401349841515862 _step‚573 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449710014797312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5853681177600 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406145978529792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395517052717056 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59238630 % _timestamp‚1.7884335851156497e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131969963 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59238628 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStP³~atsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395517035677696 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33382344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412000334086144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412000334086144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚401349170038538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚443568325750784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449710196056576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5832134360842 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚411999659707392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131990027 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131990027 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92620511 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101537502 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33381883 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233527531 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101537504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5854337553408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808Õ™ô&ìîM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5854337553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33382344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412000334086144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412000334086144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚401349170038538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚443568325750784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449710196056576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5832134360842 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚411999659707392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131990027 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131990027 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92620511 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101537502 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33381883 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233527531 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101537504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5854337553408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233507465 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101537504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚406145978529792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233527531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131969963 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚401349841515862 _step‚573 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449710014797312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5853681177600 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406145978529792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395517052717056 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59238630 % _timestamp‚1.7884335851156497e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131969963 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59238628 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395517035677696 F~–-ñ‚(:¢ »©åÔø»‘ disk.sda3.in‚0 disk.drbd1777.in‚0# disk.dm-0.out‚30776.64208984375 disk.nvme0n1.in‚ 601.26953125& disk.nvme0n1.out‚30776.64208984375" disk.nvme0c0n1.in‚ 601.26953125* proc.memory.percent‚17.531919479370117 disk.sda1.out‚0 disk.rbd1.in‚0 network.sent‚ 1.6071044e+07 disk.rbd2.in‚0 disk.drbd1777.out‚0 cpu‚25.32786504503302 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 947.90234375" proc.memory.rssMB‚ 2872.4296875 disk.sda3.out‚ 206.359375% disk./.usageGB‚1100.7999687194824 disk.dm-0.in‚ 601.26953125 disk.sda1.in‚0 disk.rbd1.out‚ 2.759765625 disk.sda2.out‚0 disk.rbd3.out‚0) disk./.usagePercent‚73.87560953345842 disk.sda.in‚0 disk.dm-1.out‚ 206.359375( disk.nvme0c0n1.out‚30776.64208984375! disk.rbd2.out‚152.07958984375 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 1.872727e+06 disk.sda.out‚ 206.359375 disk.rbd0.out‚0$ memory_percent‚94.21446323394775Á¬‘ò‚(:† Ê©åÔë—ë gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚101.73' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êNø2ó‚(:§ Ê©åÔ§ƒ‹ network.sent‚ 1.6178944e+07 disk.sda.in‚0 disk.sda3.out‚ 207.12109375 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 601.26953125 disk.drbd1777.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚73.87559314897624% disk./.usageGB‚1100.7997245788574 disk.dm-1.out‚ 207.12109375 disk.rbd1.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚30875.79345703125' proc.memory.availableMB‚ 947.8515625" proc.memory.rssMB‚ 2872.4296875 network.recv‚ 1.878852e+06 disk.rbd1.out‚ 2.759765625( disk.nvme0c0n1.out‚30875.79345703125 disk.sda3.in‚0 disk.nvme0n1.in‚ 601.26953125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 601.26953125* proc.memory.percent‚17.531919479370117 proc.cpu.threads‚17 disk.sda.out‚ 207.12109375 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0$ memory_percent‚94.21477317810059 disk.drbd1777.out‚0& disk.nvme0n1.out‚30875.79345703125 disk.sda2.in‚0! disk.rbd2.out‚152.07958984375 cpu‚25.32709078990497 disk.dm-1.in‚0ejec2ô- _wandbruntime‚5303  _runtime‚5303Fq.Äõ¾ trainer/global_step‚7049 $ _timestamp‚1.788433619233331e+09 _runtime‚5303.938099294 _step‚575 % lr-AdamW/pg1‚0.000100270489341551 % lr-AdamW/pg2‚0.000100270489341551)ƒ«üÉöà % lr-AdamW/pg2‚0.000100270489341551 trainer/global_step‚7049 $ _timestamp‚1.788433619233331e+09 _runtime‚5303.938099294 _step‚575 % lr-AdamW/pg1‚0.000100270489341551¿ü)2÷- _wandbruntime‚5303  _runtime‚5303&$~X¸#øÞO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚408907277433344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102224288 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33604164 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452773394316800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚414801124988928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93243024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5872153265286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚446589211896320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414801124988928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414801799354368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59638860 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 5303.93904781 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398207338075136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚398207355114496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884336192341099e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102224286 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166592512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5894503918080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446589187956224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132897599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132897599 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59638862 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚408907277433344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚408907295436288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132877561 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5893847555584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 142652416 trainer/global_step‚7049 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235121887 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928ÑiŒåé'ùãO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚408907277433344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102224288 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33604164 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452773394316800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚414801124988928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93243024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5872153265286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚446589211896320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414801124988928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414801799354368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59638860 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 5303.93904781 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398207338075136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚398207355114496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884336192341099e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102224286 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166592512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5894503918080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446589187956224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132897599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132897599 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59638862 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚408907277433344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚408907295436288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132877561 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5893847555584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 142652416 trainer/global_step‚7049 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235121887 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235134317 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93255624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5304.603882413 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102238304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33608596 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408963630472192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6184887536128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132896015 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33609041 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59647030 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414858297455616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235154383 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚398262242205696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414858971834368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408963648475136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235134317 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚452835917975040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446650838693888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452835726230016 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132916079 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93256071 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6185055341056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5873624138214 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33609041 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59647030 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414858297455616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235154383 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚398262242205696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414858971834368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408963648475136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235134317 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚452835917975040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446650838693888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452835726230016 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132916079 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93256071 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6185055341056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5873624138214 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235134317 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93255624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5304.603882413 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102238304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33608596 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408963630472192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6184887536128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132896015 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417659762737152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚449671724839424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102925088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411724947378688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93879598 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5935489723904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417659762737152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102925088 % _timestamp‚1.7884336540571668e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455898825288704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6227264073216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚580 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133823651 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60047260 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚449671748779520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60047262 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5912988604694 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133803613 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236748739 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚406865533207830 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102925086 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5934833361408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411724929375744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417660437102592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5935489723904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417659762737152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚449671724839424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102925088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411724947378688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93879598 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5935489723904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417659762737152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102925088 % _timestamp‚1.7884336540571668e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455898825288704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6227264073216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚580 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133823651 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60047260 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚449671748779520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60047262 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5912988604694 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133803613 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236748739 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚406865533207830 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102925086 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5934833361408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411724929375744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417660437102592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5935489723904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93892367 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚455961342809088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417717609582592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚411781282414592 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133822067 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6227967232000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401007448733696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60055430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102939102 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60055428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚455961526165504 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102939104 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚406921925250678 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5935652789248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102939104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚401007465773056 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚581 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236781235 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236761169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133842131 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236781235 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417716935203840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚449733375577088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133842131 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚406921253773354 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133822067 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6228126648320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411781300417536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417717609582592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33836939 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5936309165056 JBDeviceStatsMonitor.on_train_batch_end/segmen’—åv t.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5935652789248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 trainer/global_step‚7099 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33837387 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5913805039658 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93892817 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5914459477622 % _timestamp‚1.7884336547126215e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚411781300417536 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414i\%aô&ŒîM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚411781282414592 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102939102 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚406921925250678 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5935652789248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102939104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚401007465773056 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚581 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236781235 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236761169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133842131 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236781235 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417716935203840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚449733375577088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133842131 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚406921253773354 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133822067 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6228126648320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411781300417536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417717609582592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33836939 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5936309165056 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5935652789248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 trainer/global_step‚7099 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33837387 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5913805039658 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93892817 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5914459477622 % _timestamp‚1.7884336547126215e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚411781300417536 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93892367 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚455961342809088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417717609582592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚411781282414592 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133822067 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6227967232000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401007448733696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60055430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102939102 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60055428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚455961526165504 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102939104 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420518400485376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5953823944102 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238375591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5975819167232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5976475529728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414542599321088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34060716 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414542581318144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420519074850816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚452754285662720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103625888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5373.603820409 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103625886 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5976475529728 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5975819167232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6270159048704 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94516819 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928©‹ùf›äO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60455660 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238355551 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚414542581318144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134729665 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103625888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420518400485376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5953823944102 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238375591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5975819167232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5976475529728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414542599321088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34060716 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414542581318144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420519074850816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚452754285662720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103625888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5373.603820409 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103625886 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5976475529728 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5975819167232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6270159048704 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94516819 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134749703 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚459024612529664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116È„ ×C2œ- _wandbruntime‚5374  _runtime‚5374$ã@œï&éM PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60463828 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6271014211072 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134748119 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420576247330816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5954640379066 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884336897713253e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103639902 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403752655261696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409707967118086 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚459086926671360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚414598934356992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94529564 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5374.476566524 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103639902 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103639904 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134768183 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134768183 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238408087 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚414598952359936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94529101 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238408087 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚452815936400384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚414598934356992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5955294817030 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5976638595072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134748119 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚459087120513536 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚409707295640762 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452815912460288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6271184113152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5977294970880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚420575572952064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚414598952359936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34065734 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7149 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5977294970880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1Q&Íô&žîM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5977294970880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚420575572952064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚414598952359936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34065734 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7149 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5977294970880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60463828 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6271014211072 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134748119 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420576247330816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5954640379066 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884336897713253e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103639902 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403752655261696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409707967118086 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚459086926671360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚414598934356992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94529564 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5374.476566524 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103639902 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103639904 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134768183 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134768183 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238408087 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚414598952359936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94529101 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238408087 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚452815936400384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚414598934356992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5955294817030 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5976638595072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134748119 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚459087120513536 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚409707295640762 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452815912460288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6271184113152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430ÉðzN 2Ÿ- _wandbruntime‚5374  _runtime‚5374Uâb’X Ò 1loss/FBt4/assignment_loss‚0.11815725515286128 -loss/FBt2/detection_loss‚2.61416981741786 1loss/SRqqt1/detection_loss‚0.8938398581857865 1loss/SRqqt4/detection_loss‚1.8429436714134433 _runtime‚5374.477637211 0loss/FRt1/assignment_loss‚1.1838252913951874 0loss/FRt2/assignment_loss‚3.2274615594318936 1loss/FBt1/assignment_loss‚0.10377429534816268 .loss/FBt1/detection_loss‚0.535047148222297 %loss/total_loss‚42.10442833163685 _step‚586 /loss/FRt1/detection_loss‚1.6206742323935033 /loss/FRt4/detection_loss‚2.0142181837091258 trainer/global_step‚7149 /loss/FRt3/assignment_loss‚4.150864744186402 1loss/SRqqt3/assignment_loss‚1.284403075526158 1loss/FBt2/assignment_loss‚0.08588413252255123 0loss/FBt3/assignment_loss‚0.5117993462232074 1loss/SRqqt2/detection_loss‚1.7967573535077426 /loss/FRt2/detection_loss‚2.3253185913270835 /loss/FRt3/detection_loss‚3.0918002843856813 -loss/FBt4/detection_loss‚2.49595312277476 2loss/SRqqt1/assignment_loss‚0.9628665018970004 1loss/SRqqt3/detection_loss‚2.7938964078823725 epoch‚5 .loss/FRt4/assignment_loss‚2.54753379611408 2loss/SRqqt2/assignment_loss‚1.4257496698778502 2loss/SRqqt4/assignment_loss‚1.5628824848999185 .loss/FBt3/detection_loss‚2.914607507841928 % _timestamp‚1.7884336897729897e+09ˆT¶·]¡× % _timestamp‚1.7884336897729897e+09 1loss/FBt4/assignment_loss‚0.11815725515286128 -loss/FBt2/detection_loss‚2.61416981741786 1loss/SRqqt1/detection_loss‚0.8938398581857865 1loss/SRqqt4/detection_loss‚1.8429436714134433 _runtime‚5374.477637211 0loss/FRt1/assignment_loss‚1.1838252913951874 0loss/FRt2/assignment_loss‚3.2274615594318936 1loss/FBt1/assignment_loss‚0.10377429534816268 .loss/FBt1/detection_loss‚0.535047148222297 %loss/total_loss‚42.10442833163685 _step‚586 /loss/FRt1/detection_loss‚1.6206742323935033 /loss/FRt4/detection_loss‚2.0142181837091258 trainer/global_step‚7149 /loss/FRt3/assignment_loss‚4.150864744186402 1loss/SRqqt3/assignment_loss‚1.284403075526158 1loss/FBt2/assignment_loss‚0.08588413252255123 0loss/FBt3/assignment_loss‚0.5117993462232074 1loss/SRqqt2/detection_loss‚1.7967573535077426 /loss/FRt2/detection_loss‚2.3253185913270835 /loss/FRt3/detection_loss‚3.0918002843856813 -loss/FBt4/detection_loss‚2.49595312277476 2loss/SRqqt1/assignment_loss‚0.9628665018970004 1loss/SRqqt3/detection_loss‚2.7938964078823725 epoch‚5 .loss/FRt4/assignment_loss‚2.54753379611408 2loss/SRqqt2/assignment_loss‚1.4257496698778502 2loss/SRqqt4/assignment_loss‚1.5628824848999185 .loss/FBt3/detection_loss‚2.914607507841928ÊÞµ¸ž¢‚(:“ ¤ªåÔ×ý gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.152' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.830399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01Úê­(£‚(: ¤ªåÔ¯ï‘ disk.dm-0.in‚ 601.26953125 disk.sda1.in‚0 disk.rbd1.out‚ 2.759765625" disk.nvme0c0n1.in‚ 601.26953125( disk./.usagePercent‚73.8756699512365 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0$ proc.memory.availableMB‚947.6875" proc.memory.rssMB‚ 2872.4296875$ disk./.usageGB‚1100.800868988037 disk.sda2.out‚0 disk.sda3.in‚0" disk.dm-0.out‚31372.8212890625 disk.sda.in‚0 disk.sda.out‚ 210.61328125* proc.memory.percent‚17.531919479370117 network.sent‚ 1.6411765e+07 disk.rbd3.in‚0 cpu‚25.32237954550873 network.recv‚ 1.901815e+06 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 153.919921875 disk.sda3.out‚ 210.61328125 disk.nvme0n1.in‚ 601.26953125% disk.nvme0n1.out‚31372.8212890625' disk.nvme0c0n1.out‚31372.8212890625 disk.drbd1777.out‚0 disk.dm-1.in‚0$ memory_percent‚94.21577453613281 disk.rbd0.in‚0 disk.dm-1.out‚ 210.61328125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1777.in‚0:nawt¤‚(:† ³ªåԽƑŽ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.26' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.K<+›0.uncorrectedMemoryErrors‚0oAt|/¥‚(:¤ ³ªåÔÏœ¹ disk.rbd2.out‚ 155.650390625% disk./.usageGB‚1100.8008422851562 disk.sda.in‚0( disk.nvme0c0n1.out‚31471.98193359375 cpu‚25.321633050755505 network.sent‚ 1.6519002e+07 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 601.26953125* proc.memory.percent‚17.531919479370117 network.recv‚ 1.906374e+06 disk.rbd3.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0$ memory_percent‚94.21889781951904( proc.memory.availableMB‚ 947.17578125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 601.26953125& disk.nvme0n1.out‚31471.98193359375 disk.rbd1.in‚0 disk.rbd1.out‚ 2.759765625 disk.sda2.in‚0 disk.sda3.out‚ 211.1015625 disk.drbd1777.in‚0# disk.dm-0.out‚31471.98193359375 disk.sda.out‚ 211.1015625 disk.nvme0n1.in‚ 601.26953125 disk.sda2.out‚0" proc.memory.rssMB‚ 2872.4296875 disk.drbd1777.out‚0 disk.dm-1.out‚ 211.1015625 disk.rbd0.in‚0) disk./.usagePercent‚73.87566815918376 proc.cpu.threads‚17¡ðÉ2¦- _wandbruntime‚5408  _runtime‚5408Ú-ÌǧÁ & lr-AdamW/pg1‚9.540951184167002e-05 & lr-AdamW/pg2‚9.540951184167002e-05 trainer/global_step‚7199 % _timestamp‚1.7884337238989313e+09 _runtime‚5408.603721593 _step‚587*œ0Ì¨Æ _runtime‚5408.603721593 _step‚587 & lr-AdamW/pg1‚9.540951184167002e-05 & lr-AdamW/pg2‚9.540951184167002e-05 trainer/global_step‚7199 % _timestamp‚1.7884337238989313e+09ËD¿}r2©- _wandbruntime‚5408  _runtime‚5408R‰Ô å'ªßO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417360251263488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417360233260544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406442974698496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5995313721266 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚412438288419762 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423377712599040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239982403 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135675755 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _step‚588 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34288937 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462150037248512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5408.604557968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423377038233600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240002443 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6313238642688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135655717 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417360233260544 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406442974698496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5995313721266 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚412438288419762 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423377712599040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239982403 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135675755 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _step‚588 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34288937 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462150037248512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5408.604557968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423377038233600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240002443 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6313238642688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135655717 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417360233260544 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417360251263488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928Ì-¶¿2¬- _wandbruntime‚5409  _runtime‚5409&jÃï&­éM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60872230 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423434885079040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34293432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚455898473283584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6017624400896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884337245617044e+09 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104340702 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95166106 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60872228 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6314274786816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚455898449343488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5995475718474 trainer/global_step‚7199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417416604302336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚462212568908288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417416604302336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95165660 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5996130156438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417416586299392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104340704 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135674171 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135674171 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240034939 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417416586299392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240034939 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6314119564800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚406497861789696 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135694235 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240014873 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34293876 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104340702 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462212748070400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚412494008985494 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417416604302336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95165660 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5996130156438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417416586299392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104340704 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135674171 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135674171 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240034939 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417416586299392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240034939 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6314119564800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚406497861789696 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135694235 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240014873 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34293876 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104340702 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462212748070400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚412494008985494 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104340702 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95166106 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60872228 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6314274786816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0Í|Z/õ2¯- _wandbruntime‚5409  _runtime‚5409Jq³Å]°× /loss/FRt4/assignment_loss‚3.028752202987671 1loss/FBt1/assignment_loss‚0.09777469779266831 % _timestamp‚1.7884337245634909e+09 1loss/SRqqt4/assignment_loss‚1.556858445538415 3loss/FBt2/assignment_loss‚0.0009838866414308238 /loss/FRt3/detection_loss‚2.7377213585761284 1loss/SRqqt2/detection_loss‚1.9711518725182147 2loss/SRqqt2/assignment_loss‚1.0743691049357678 .loss/FBt4/detection_loss‚2.415489415327708 _runtime‚5409.268126134 0loss/FRt2/assignment_loss‚2.8727482907150104 /loss/FRt3/assignment_loss‚3.510082556355384 2loss/SRqqt3/assignment_loss‚2.1346072095135846 /loss/FBt3/detection_loss‚2.5954480171203613 1loss/FBt3/assignment_loss‚0.17233422414089242 .loss/FRt4/detection_loss‚2.325109241604805 1loss/SRqqt3/detection_loss‚2.5785398731629052 /loss/FRt2/detection_loss‚2.3840139359235764 1loss/FBt4/assignment_loss‚0.31466469044486683 /loss/FBt1/detection_loss‚0.5634736547470782 trainer/global_step‚7199 2loss/SRqqt1/assignment_loss‚0.9473509446789439 1loss/SRqqt1/detection_loss‚0.8450467479534638 .loss/FBt2/detection_loss‚3.826024909814199 _step‚590 /loss/FRt1/assignment_loss‚1.643862523594681 /loss/FRt1/detection_loss‚1.7591054684729601 0loss/SRqqt4/detection_loss‚2.092554920911789 %loss/total_loss‚43.44806819347251 epoch‚5Sfg b±Ü % _timestamp‚1.7884337245634909e+09 1loss/SRqqt4/assignment_loss‚1.556858445538415 3loss/FBt2/assignment_loss‚0.0009838866414308238 /loss/FRt3/detection_loss‚2.7377213585761284 1loss/SRqqt2/detection_loss‚1.9711518725182147 2loss/SRqqt2/assignment_loss‚1.0743691049357678 .loss/FBt4/detection_loss‚2.415489415327708 _runtime‚5409.268126134 0loss/FRt2/assignment_loss‚2.8727482907150104 /loss/FRt3/assignment_loss‚3.510082556355384 2loss/SRqqt3/assignment_loss‚2.1346072095135846 /loss/FBt3/detection_loss‚2.5954480171203613 1loss/FBt3/assignment_loss‚0.17233422414089242 .loss/FRt4/detection_loss‚2.325109241604805 1loss/SRqqt3/detection_loss‚2.5785398731629052 /loss/FRt2/detection_loss‚2.3840139359235764 1loss/FBt4/assignment_loss‚0.31466469044486683 /loss/FBt1/detection_loss‚0.5634736547470782 trainer/global_step‚7199 2loss/SRqqt1/assignment_loss‚0.9473509446789439 1loss/SRqqt1/detection_loss‚0.8450467479534638 .loss/FBt2/detection_loss‚3.826024909814199 _step‚590 /loss/FRt1/assignment_loss‚1.643862523594681 /loss/FRt1/detection_loss‚1.7591054684729601 0loss/SRqqt4/detection_loss‚2.092554920911789 %loss/total_loss‚43.44806819347251 epoch‚5 /loss/FRt4/assignment_loss‚3.028752202987671 1loss/FBt1/assignment_loss‚0.09777469779266831θĦ²‚(:’ ªåÔÜ܉Ž gpu.0.gpu‚44.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚92.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.470599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?0˜0³‚(:¥ ªåÔ”¹¼‘* proc.memory.percent‚17.531919479370117 disk.drbd1777.in‚0 disk.sda.in‚0 cpu‚25.3208284530045 network.sent‚ 1.6524242e+07) disk./.usagePercent‚73.87566815918376 disk.drbd1777.out‚0 disk.dm-1.out‚ 212.01953125 disk.nvme0n1.in‚ 601.26953125" disk.nvme0c0n1.in‚ 601.26953125 disk.sda2.in‚0 disk.sda.out‚ 212.01953125 disk.rbd3.in‚0 disk.dm-0.in‚ 601.26953125 disk.rbd2.out‚ 155.650390625% disk./.usageGB‚1100.8008422851562 disk.rbd1.in‚0( disk.nvme0c0n1.out‚31472.61865234375 disk.sda2.out‚0 disk.rbd0.out‚0$ memory_percent‚94.21513080596924 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0& disk.nvme0n1.out‚31472.61865234375 disk.rbd1.out‚ 2.759765625" proc.memory.rssMB‚ 2872.4296875# disk.dm-0.out‚31472.61865234375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0( proc.memory.availableMB‚ 947.79296875 proc.cpu.threads‚17 network.recv‚ 1.910119e+06 disk.sda3.out‚ 212.01953125 disk.dm-1.in‚0\²T´‚(:’ ѪåÔáÉŽ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚90.039' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.007800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ)1µ‚(:¦ ѪåÔëí¥ disk.rbd1.in‚0( proc.memory.availableMB‚ 947.51953125 disk.sda3.out‚ 212.3828125 disk.rbd1.out‚ 2.759765625" disk.nvme0c0n1.in‚ 601.26953125 disk.rbd0.in‚0% disk./.usageGB‚1100.8008422851562 disk.sda.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 601.26953125 disk.sda3.in‚0& disk.nvme0n1.out‚31571.70849609375( disk.nvme0c0n1.out‚31571.70849609375 disk.rbd0.out‚0" proc.memory.rssMB‚ 2872.4296875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.dm-1.out‚ 212.3828125* proc.memory.percent‚17.531919479370117 network.recv‚ 1.915044e+06 disk.rbd3.in‚0 disk.sda.out‚ 212.3828125 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚94.21679973602295 network.sent‚ 1.6631917e+07) disk./.usagePercent‚73.87566815918376 disk.rbd3.out‚0# disk.dm-0.out‚31571.70849609375 cpu‚25.320095129743244 disk.dm-0.in‚ 601.26953125 disk.dm-1.in‚0 disk.sda2.out‚0! disk.rbd2.out‚155.81103515625 disk.drbd1777.in‚0 disk.drbd1777.out‚0“Lz‚2¶- _wandbruntime‚5443  _runtime‚5443Bj¹Æ·À & lr-AdamW/pg1‚9.379619790776801e-05 & lr-AdamW/pg2‚9.379619790776801e-05 trainer/global_step‚7249 $ _timestamp‚1.788433758812854e+09 _runtime‚5443.517594067 _step‚591Cí\˸Š_step‚591 & lr-AdamW/pg1‚9.379619790776801e-05 & lr-AdamW/pg2‚9.379619790776801e-05 trainer/global_step‚7249 $ _timestamp‚1.788433758812854e+09 _runtime‚5443.517594067Ïvà÷92¹- _wandbruntime‚5443  _runtime‚5443©7ÿ'å'ºßO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458919359429120 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136581769 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6356329098240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6036149060674 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426235675981824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6057790778880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6057790778880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚420177885202944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚409188181226496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458919335489024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241609255 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚420177903205888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105027486 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _runtime‚5443.518481809 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6058447141376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚426236350347264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105027488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426236350347264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 % _timestamp‚1.7884337588135796e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426235675981824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465275845859840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465275664587264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 trainer/global_step‚7249 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415224330287170 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105027486 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61272460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415223658810054 _step‚592 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415224330287170 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105027486 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61272460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415223658810054 _step‚592 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426235675981824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6057790778880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6057790778880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚420177885202944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚409188181226496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458919335489024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241609255 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚420177903205888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105027486 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _runtime‚5443.518481809 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6058447141376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactiveañU²& _split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚426236350347264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105027488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426236350347264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 % _timestamp‚1.7884337588135796e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426235675981824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465275845859840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465275664587264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 trainer/global_step‚7249ÐÚJ|™2¼- _wandbruntime‚5444  _runtime‚5444önµî&½èM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105041504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136620287 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 $ _timestamp‚1.788433759467092e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6036311057882 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241661791 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6058610206720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚7249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34521084 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61280630 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241641725 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458981010166784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95801272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95801714 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241661791 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚415280050852902 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚426293522827264 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚420234238241792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136600223 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409243085357056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚420234256244736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6059266582528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105041502 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚465338168358400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚426292848448512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61280628 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6357352033792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚420234238241792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚465338362200576 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚409243068317696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚593 _runtime‚5444.172238852 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0àªÒó&¾íM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6059266582528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241661791 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6058610206720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚7249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34521084 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61280630 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241641725 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458981010166784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95801272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95801714 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241661791 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚415280050852902 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚426293522827264 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚420234238241792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136600223 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409243085357056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚420234256244736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6059266582528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105041502 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚465338168358400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚426292848448512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61280628 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6357352033792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚420234238241792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚465338362200576 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚409243068317696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚593 _runtime‚5444.172238852 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105041504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136620287 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 $ _timestamp‚1.788433759467092e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6036311057882 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 242677869 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6079955823104 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 105627062 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚411192096094208 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚428344277631488 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 6079955823104 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚422264321808384 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6080448653312 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 141381120 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 6378862983168 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6057571833338 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚422264339811328 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 105627060 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 6058062747730 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚428344788464640 _step‚595 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 137034830 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6080448653312 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0üªÂ•Q*ÌËT hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 6378862983168 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6057571833338 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚422264339811328 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 105627060 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 6058062747730 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚428344788464640 _step‚595 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 137034830 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6080448653312 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 trainer/global_step‚7253 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚428344788464640 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 137050807 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚461083500119040 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 105627062 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884338084669707e+09 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚422264339811328 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚96393522 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚34761373 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 6378745542144 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 105627060 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 242677869 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚96393369 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚461083524059136 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚151 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚417249650888186 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 242677869 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6079955823104 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 105627062 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚411192096094208 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚428344277631488 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 6079955823104 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚422264321808384 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6080448653312 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 141381120Ó&Œ†v2Í- _wandbruntime‚5494  _runtime‚5494¨Lu³V)ÎÐR G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚467500627902464 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 117441024 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 242679998 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚417285308257082 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6058426357138 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚422301169840128 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 105637868 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚422301169840128 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚96404272 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚34765772 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚467500486521344 % _timestamp‚1.7884338093660104e+09 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚428381472786944 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 242679998 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 6080320949760 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚417285816210834 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6379110668800 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 6057935442746 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 137042132 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6379228109824 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 _runtime‚5494.071127454ô»ò?ÐÏÕR [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 105637868 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚422301169840128 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚96404272 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚34765772 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚467500486521344 % _timestamp‚1.7884338093660104e+09 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚428381472786944 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 242679998 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 6080320949760 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚417285816210834 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6379110668800 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 6057935442746 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 137042132 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6379228109824 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 _runtime‚5494.071127454 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚467500627902464 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 117441024 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 242679998 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚417285308257082 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6058426357138 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚422301169840128 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 105637868 B&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.41228070855140686 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38790053129196167 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.596875011920929 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5122088193893433 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9705459475517273 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 particle/accuracy_5_of_7‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961832761764526 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5236924886703491 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5055090188980103 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.49206212162971497 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚7253 _step‚597 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17222344875335693 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1301398128271103 particle/accuracy_2_of_2‚1 jet/accuracy_7_of_7‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5521110892295837 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6203703880310059 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9860575199127197 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5181189775466919 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +jet/accuracy_4_of_5‚0.00994623638689518 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29180270433425903 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3392857015132904 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.526190459728241 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24074074625968933 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.525068998336792 5 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.45681032538414 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987546801567078 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9871838688850403 particle/accuracy_4_of_4‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5524173974990845 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4608237147331238 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.40286192297935486 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5365410447120667 particle/accuracy_2_of_6‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 epoch‚5 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023036666214466095 +particle/specificity‚0.9527902603149414 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 *loss/val_total_loss‚43.043920409185944 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06028369069099426 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.526161789894104 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_2‚0.7671517729759216 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.03333333507180214 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26688456535339355 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_1_of_1‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10163531452417374 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013188950717449188 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_6‚1 particle/accuracy_6_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21906578540802002 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03362203761935234 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01999177411198616 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2214927077293396 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9889131188392639 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13539308309555054 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1641400009393692 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5336198806762695 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2664228677749634 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04494047537446022 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 jet/accuracy_3_of_6‚0.25 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5359869599342346 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07864223420619965 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.37037041783332825 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6032509207725525 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15333932638168335 +jet/accuracy_2_of_4‚0.39278867840766907 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 particle/accuracy_3_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1708521693944931 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 particle/accuracy_2_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9825456738471985 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17875182628631592 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5794106125831604 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.500325620174408 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3541666865348816 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.543029248714447 ,jet/accuracy_4_of_4‚0.004681502003222704 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.640350878238678 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43048349022865295 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9590566158294678 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4898819625377655 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4637930989265442 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1350574791431427 *jet/accuracy_1_of_1‚0.5021185874938965 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19441306591033936 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9824561476707458 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2983561158180237 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2541152834892273 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.029069768264889717 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07959906756877899 jet/accuracy_5_of_5‚0 jet/accuracy_1_of_7‚1 % _timestamp‚1.7884338215968115e+09 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10921800881624222 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010204082354903221 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.46482551097869873 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15366333723068237 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15510793030261993 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9735791683197021 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.007377048954367638 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.500116765499115 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_5_of_6‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.817503809928894 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_3_of_4‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26390954852104187 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16405977308750153 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20188307762145996 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.5320513248443604 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.983270525932312 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.016509195789694786 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9042553305625916 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4105859696865082 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRÁ2"›4?t4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5210474729537964 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 _runtime‚5506.303518143 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6006289124488831 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17677786946296692 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.984610378742218 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13884010910987854 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09944798052310944 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.989690363407135 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20604056119918823 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5505816340446472 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,"Purity/3FRt0SRqqt1FBt/event_purity‚0.125 jet/accuracy_5_of_7‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2338361293077469 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1951146423816681 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3517361581325531 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19026432931423187 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5177438855171204 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2711186707019806 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1454917937517166 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39909908175468445 +jet/accuracy_3_of_3‚0.02705606259405613 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25731146335601807 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8728070259094238 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4874325096607208 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5453947186470032 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5446348786354065 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6445481181144714 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5144099593162537 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.991970419883728 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13065844774246216 *jet/accuracy_1_of_4‚0.9228556156158447 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.979095458984375 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5110512375831604 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008217359893023968 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1665254384279251 (particle/accuracy‚0.8525391817092896 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5218819379806519 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4675382077693939 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5000661015510559 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.49734675884246826 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5359869599342346 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.517677366733551 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20212070643901825 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16214843094348907 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9799411296844482 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5448679327964783 ,particle/sensitivity‚0.31073927879333496 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.479166716337204 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5008028149604797 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5476446747779846 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 +jet/accuracy_3_of_5‚0.08825524896383286 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5133333206176758 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16221392154693604 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9701929688453674 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16228815913200378 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21541330218315125 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01684636063873768 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.535493791103363 particle/accuracy_5_of_5‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.03333333507180214 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.988940954208374 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9868658781051636 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5414478182792664 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983104467391968 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983104467391968 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17222344875335693 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9836651086807251 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4154728055000305 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5733333826065063 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40469491481781006 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27540475130081177 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4712621867656708 particle/accuracy_3_of_5‚1 particle/accuracy_1_of_6‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9950178861618042 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3856588900089264 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 jet/accuracy_1_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006525573320686817 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4596669673919678 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5479418635368347 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06454980373382568 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.18888889253139496 particle/accuracy_4_of_7‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29510971903800964 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16219764947891235 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16367073357105255 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4790849983692169 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 jet/accuracy_3_of_7‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37775298953056335 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5786449909210205 &particle/f_score‚0.396647572517395 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6966373324394226 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0674603208899498 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_6_of_6‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5855222940444946 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.028106775134801865 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_2_of_4‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4385325312614441 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.46764126420021057 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_1_of_3‚0.876510739326477 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2117406576871872 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0415295735001564 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.985295295715332 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06500257551670074 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166667461395264 particle/accuracy_1_of_7‚1 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5445402264595032 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14536243677139282 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.06081029027700424 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5610615015029907 6validation_average_jet_accuracy‚0.4475039541721344 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9874097108840942 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05ršùD¾@ÔŽ§ @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5446348786354065 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6445481181144714 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5144099593162537 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.991970419883728 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13065844774246216 *jet/accuracy_1_of_4‚0.9228556156158447 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.979095458984375 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5110512375831604 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008217359893023968 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1665254384279251 (particle/accuracy‚0.8525391817092896 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5218819379806519 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4675382077693939 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5000661015510559 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.49734675884246826 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5359869599342346 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.517677366733551 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20212070643901825 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16214843094348907 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9799411296844482 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5448679327964783 ,particle/sensitivity‚0.31073927879333496 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.479166716337204 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5008028149604797 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5476446747779846 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 +jet/accuracy_3_of_5‚0.08825524896383286 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5133333206176758 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16221392154693604 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9701929688453674 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16228815913200378 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21541330218315125 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01684636063873768 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.535493791103363 particle/accuracy_5_of_5‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.03333333507180214 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.988940954208374 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9868658781051636 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5414478182792664 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983104467391968 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983104467391968 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17222344875335693 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9836651086807251 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4154728055000305 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5733333826065063 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40469491481781006 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27540475130081177 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4712621867656708 particle/accuracy_3_of_5‚1 particle/accuracy_1_of_6‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9950178861618042 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3856588900089264 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 jet/accuracy_1_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006525573320686817 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4596669673919678 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5479418635368347 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06454980373382568 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.18888889253139496 particle/accuracy_4_of_7‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29510971903800964 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16219764947891235 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16367073357105255 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4790849983692169 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 jet/accuracy_3_of_7‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37775298953056335 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5786449909210205 &particle/f_score‚0.396647572517395 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6966373324394226 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0674603208899498 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_6_of_6‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5855222940444946 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.028106775134801865 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_2_of_4‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4385325312614441 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.46764126420021057 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_1_of_3‚0.876510739326477 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2117406576871872 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0415295735001564 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.985295295715332 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06500257551670074 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166667461395264 particle/accuracy_1_of_7‚1 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5445402264595032 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14536243677139282 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.06081029027700424 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5610615015029907 6validation_average_jet_accuracy‚0.4475039541721344 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9874097108840942 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9880040287971497 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18781892955303192 jet/accuracy_6_of_7‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28130531311035156 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15409356355667114 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3598671555519104 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34269270300865173 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3638889193534851 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24747473001480103 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606062412261963 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3474152684211731 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5850806832313538 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5177927613258362 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909090638160706 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.11227422207593918 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2522522509098053 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9966133832931519 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2692528963088989 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36569923162460327 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2545120120048523 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9966883659362793 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.26018521189689636 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0028964863158762455 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.33611106872558594 jet/accuracy_5_of_6‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.493055522441864 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16206897795200348 *jet/accuracy_2_of_2‚0.1135304719209671 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4259259104728699 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5409452319145203 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5773580074310303 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.523809552192688 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15395475924015045 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.971230149269104 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.41228070855140686 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38790053129196167 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_propoSžº·×Rrtion‚0.00012400794366840273 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.596875011920929 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5122088193893433 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9705459475517273 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 particle/accuracy_5_of_7‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961832761764526 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5236924886703491 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5055090188980103 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.49206212162971497 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚7253 _step‚597 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17222344875335693 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1301398128271103 particle/accuracy_2_of_2‚1 jet/accuracy_7_of_7‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5521110892295837 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6203703880310059 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9860575199127197 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5181189775466919 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +jet/accuracy_4_of_5‚0.00994623638689518 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29180270433425903 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3392857015132904 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.526190459728241 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24074074625968933 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.525068998336792 5 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.45681032538414 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987546801567078 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9871838688850403 particle/accuracy_4_of_4‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5524173974990845 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4608237147331238 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.40286192297935486 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5365410447120667 particle/accuracy_2_of_6‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 epoch‚5 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023036666214466095 +particle/specificity‚0.9527902603149414 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 *loss/val_total_loss‚43.043920409185944 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06028369069099426 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.526161789894104 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_2‚0.7671517729759216 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.03333333507180214 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26688456535339355 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_1_of_1‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10163531452417374 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013188950717449188 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_6‚1 particle/accuracy_6_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21906578540802002 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03362203761935234 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01999177411198616 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2214927077293396 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9889131188392639 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13539308309555054 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1641400009393692 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5336198806762695 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2664228677749634 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04494047537446022 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 jet/accuracy_3_of_6‚0.25 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5359869599342346 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07864223420619965 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.37037041783332825 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6032509207725525 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15333932638168335 +jet/accuracy_2_of_4‚0.39278867840766907 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 particle/accuracy_3_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1708521693944931 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 particle/accuracy_2_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9825456738471985 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17875182628631592 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5794106125831604 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.500325620174408 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3541666865348816 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.543029248714447 ,jet/accuracy_4_of_4‚0.004681502003222704 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.640350878238678 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43048349022865295 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9590566158294678 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4898819625377655 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4637930989265442 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1350574791431427 *jet/accuracy_1_of_1‚0.5021185874938965 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19441306591033936 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9824561476707458 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2983561158180237 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2541152834892273 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.029069768264889717 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07959906756877899 jet/accuracy_5_of_5‚0 jet/accuracy_1_of_7‚1 % _timestamp‚1.7884338215968115e+09 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10921800881624222 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010204082354903221 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.46482551097869873 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15366333723068237 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15510793030261993 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9735791683197021 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.007377048954367638 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.500116765499115 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_5_of_6‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.817503809928894 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_3_of_4‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26390954852104187 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16405977308750153 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20188307762145996 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.5320513248443604 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.983270525932312 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.016509195789694786 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9042553305625916 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4105859696865082 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5210474729537964 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 _runtime‚5506.303518143 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6006289124488831 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17677786946296692 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.984610378742218 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13884010910987854 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09944798052310944 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.989690363407135 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20604056119918823 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5505816340446472 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,"Purity/3FRt0SRqqt1FBt/event_purity‚0.125 jet/accuracy_5_of_7‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2338361293077469 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1951146423816681 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3517361581325531 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19026432931423187 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5177438855171204 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2711186707019806 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1454917937517166 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39909908175468445 +jet/accuracy_3_of_3‚0.02705606259405613 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25731146335601807 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8728070259094238 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4874325096607208 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5453947186470032ÕÊ$üy’Õ‚(:‡ ««åÔâýç gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚93.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X¦y{Ö‚(:‘ ««åÔÏû”‘( disk.nvme0c0n1.out‚33035.52587890625 disk.rbd0.in‚0! disk.rbd2.out‚157.62353515625 disk.dm-0.in‚ 601.26953125 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0) proc.memory.percent‚17.53401756286621 cpu‚25.369120827477353% disk./.usageGB‚1100.8043365478516 disk.dm-1.out‚215.5 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 1424.0390625 disk.sda.in‚0# memory_percent‚91.3083553314209 proc.cpu.threads‚17 disk.sda3.out‚215.5 network.sent‚ 1.686578e+07) disk./.usagePercent‚73.87590266208512 disk.sda.out‚215.5 disk.nvme0n1.in‚ 601.26953125" disk.nvme0c0n1.in‚ 601.26953125 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 2.759765625 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2872.7734375 network.recv‚ 1.939486e+06 disk.drbd1777.out‚0# disk.dm-0.out‚33035.52587890625& disk.nvme0n1.out‚33035.52587890625™Sõï“ׂ(:ˆ º«åÔ•ŸˆŽ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.472' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.6944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pi+0Ø‚(:¥ º«åÔ¡íÕ disk.rbd1.out‚ 2.759765625 disk.drbd1777.out‚0# disk.dm-0.out‚33134.86083984375 disk.nvme0n1.in‚ 601.26953125 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda.in‚0) disk./.usagePercent‚73.87590650219813 disk.sda2.in‚0 disk.sda3.out‚ 215.7890625 disk.dm-1.in‚0 disk.dm-1.out‚ 215.7890625 network.sent‚ 1.6989269e+07( disk.nvme0c0n1.out‚33134.86083984375 disk.rbd3.in‚0) proc.memory.availableMB‚ 1424.30078125" disk.nvme0c0n1.in‚ 601.26953125 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 601.26953125 disk.sda.out‚ 215.7890625" proc.memory.rssMB‚ 2872.7734375 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0% disk./.usageGB‚1100.8043937683105& disk.nvme0n1.out‚33134.86083984375 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0$ memory_percent‚91.30675792694092) proc.memory.percent‚17.53401756286621 disk.rbd2.out‚157.6435546875 cpu‚25.368264576483952 network.recv‚ 1.945122e+06Ý Ø2Ù- _wandbruntime‚5542  _runtime‚5542rn<ÚÇÚÁ trainer/global_step‚7299 % _timestamp‚1.7884338575191994e+09 _runtime‚5542.223959755 _step‚598 & lr-AdamW/pg1‚9.218695091343162e-05 & lr-AdamW/pg2‚9.218695091343162e-05'%n“ÌÛÆ _runtime‚5542.223959755 _step‚598 & lr-AdamW/pg1‚9.218695091343162e-05 & lr-AdamW/pg2‚9.218695091343162e-05 trainer/global_step‚7299 % _timestamp‚1.7884338575191994e+09ÖA¹´2Ü- _wandbruntime‚5542  _runtime‚5542â!XÁ{ÝßO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425315828959744 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137989822 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244398888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414156877562880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚464388043582976 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420256132025288 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414156894602240 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137969784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚7299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431438299879936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6422810206208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚431438299879936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6422648679424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106409066 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106409066 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431437625514496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35027876 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425315846962688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6122452917248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431437625514496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137989822 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚425315828959744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚470810877729280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚5542.225030693 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ð°gê'ÞäO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431438299879936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6422810206208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚431438299879936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6422648679424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106409066 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106409066 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431437625514496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35027876 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425315846962688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6122452917248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431437625514496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137989822 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚425315828959744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚470810877729280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚5542.225030693 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425315828959744 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137989822 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244398888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414156877562880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚464388043582976 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420256132025288 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414156894602240 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137969784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚7299×5Î-M2ß- _wandbruntime‚5542  _runtime‚5542E,‘í&àçM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425372181998592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚470873200859648 # _timestamp‚1.78843385817883e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244431384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414211798732800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚5542.883908056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244431384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6123272358400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97131601 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106423082 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6122615982592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚425372181998592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6122615982592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚7299 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚464449718260736 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244411318 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137988238 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420311852590812 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106423082 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106423080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138008302 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35032819 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚470873388410368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97131145 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35032365 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6423506539008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464449694320640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244411318 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431495472359936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6123272358400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6423670149632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420312524068136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62098782 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚7299 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚464449718260736 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244411318 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137988238 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420311852590812 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106423082 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106423080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138008302 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35032819 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚470873388410368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97131145 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35032365 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6423506539008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464449694320640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244411318 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431495472359936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6123272358400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6423670149632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420312524068136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62098782 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244431384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414211798732800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚5542.883908056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244431384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6123272358400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97131601 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106423082 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ØqXÐA2â- _wandbruntime‚5542  _runtime‚5542$Ò~é8ãÓ 0loss/FRt4/assignment_loss‚3.3138499628929865 0loss/FBt4/assignment_loss‚0.3001914327698095 epoch‚6 trainer/global_step‚7299 0loss/FRt1/assignment_loss‚1.4142120212316514 0loss/FRt2/assignment_loss‚3.2606093805769216 /loss/FRt4/detection_loss‚2.1661724384341934 1loss/SRqqtØTÑ!2/detection_loss‚2.0498793765499785 -loss/FBt4/detection_loss‚2.20052969455719 _runtime‚ 5542.88515741 2loss/SRqqt1/assignment_loss‚1.0490977005227595 2loss/SRqqt2/assignment_loss‚1.4770214040134404 1loss/SRqqt4/assignment_loss‚1.267557009226746 3loss/FBt3/assignment_loss‚0.0011833013501018285 .loss/FRt3/assignment_loss‚3.41540617412991 /loss/FRt2/detection_loss‚2.2864333255135496 1loss/SRqqt4/detection_loss‚1.8881846182876163 /loss/FBt2/detection_loss‚3.2607624650001528 %loss/total_loss‚43.23209495992505 .loss/FRt3/detection_loss‚3.340993457370334 _step‚601 1loss/SRqqt1/detection_loss‚0.7637085002922529 1loss/SRqqt3/detection_loss‚2.1766065429238712 /loss/FBt3/detection_loss‚2.7715893149375916 2loss/SRqqt3/assignment_loss‚2.0825293362140656 0loss/FBt1/assignment_loss‚0.1600084288604548 0loss/FBt2/assignment_loss‚0.4279496669652872 .loss/FRt1/detection_loss‚1.475284710712731 /loss/FBt1/detection_loss‚0.6823346965914563 $ _timestamp‚1.788433858180474e+09ÌeüÎ^äØ 2loss/SRqqt1/assignment_loss‚1.0490977005227595 2loss/SRqqt2/assignment_loss‚1.4770214040134404 1loss/SRqqt4/assignment_loss‚1.267557009226746 3loss/FBt3/assignment_loss‚0.0011833013501018285 .loss/FRt3/assignment_loss‚3.41540617412991 /loss/FRt2/detection_loss‚2.2864333255135496 1loss/SRqqt4/detection_loss‚1.8881846182876163 /loss/FBt2/detection_loss‚3.2607624650001528 %loss/total_loss‚43.23209495992505 .loss/FRt3/detection_loss‚3.340993457370334 _step‚601 1loss/SRqqt1/detection_loss‚0.7637085002922529 1loss/SRqqt3/detection_loss‚2.1766065429238712 /loss/FBt3/detection_loss‚2.7715893149375916 2loss/SRqqt3/assignment_loss‚2.0825293362140656 0loss/FBt1/assignment_loss‚0.1600084288604548 0loss/FBt2/assignment_loss‚0.4279496669652872 .loss/FRt1/detection_loss‚1.475284710712731 /loss/FBt1/detection_loss‚0.6823346965914563 $ _timestamp‚1.788433858180474e+09 0loss/FRt4/assignment_loss‚3.3138499628929865 0loss/FBt4/assignment_loss‚0.3001914327698095 epoch‚6 trainer/global_step‚7299 0loss/FRt1/assignment_loss‚1.4142120212316514 0loss/FRt2/assignment_loss‚3.2606093805769216 /loss/FRt4/detection_loss‚2.1661724384341934 1loss/SRqqt2/detection_loss‚2.0498793765499785 -loss/FBt4/detection_loss‚2.20052969455719 _runtime‚ 5542.88515741Ù¢N’å‚(:‡ É«åÔÁŽŽ gpu.0.gpu‚19.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚100.785' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.157 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\5Gž.æ‚(:£ É«åÔÜèá disk.nvme0n1.in‚ 601.26953125 cpu‚25.367362698505556 proc.cpu.threads‚17 disk.rbd0.out‚0& disk.nvme0n1.out‚33135.32373046875 disk.rbd1.in‚0) proc.memory.percent‚17.53401756286621 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda1.in‚0 network.sent‚ 1.6994613e+07' disk./.usagePercent‚73.875920326605 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 216.41015625 disk.sda2.out‚0' proc.memory.availableMB‚ 1424.046875 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 601.26953125" proc.memory.rssMB‚ 2872.7734375 disk.sda3.in‚0 disk.sda2.in‚0$ memory_percent‚91.30830764770508 network.recv‚ 1.948867e+06$ disk./.usageGB‚1100.804599761963 disk.sda3.out‚ 216.41015625 disk.sda.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 601.26953125 disk.sda1.out‚0 disk.dm-1.out‚ 216.41015625 disk.rbd1.out‚ 2.908203125( disk.nvme0c0n1.out‚33135.32373046875 disk.rbd2.out‚157.6591796875# disk.dm-0.out‚33135.32373046875s²_ö’ç‚(:‡ Ø«åÔ»˜þ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.965 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ùqˆs-è‚(:¢ Ø«åÔÐÈ–) disk./.usagePercent‚73.87591904656732% disk./.usageGB‚1100.8045806884766 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.sda1.in‚0% disk.nvme0n1.out‚33235.0205078125$ memory_percent‚91.30642414093018" proc.memory.rssMB‚ 2872.7734375 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 216.7734375 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚33235.0205078125 network.sent‚ 1.7102479e+07 network.recv‚ 1.954065e+06 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd2.out‚159.2958984375 disk.sda3.out‚ 216.7734375 disk.drbd1777.out‚0 disk.dm-0.in‚ 601.26953125' disk.nvme0c0n1.out‚33235.0205078125 disk.rbd1.out‚ 2.908203125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 601.26953125) proc.memory.availableMB‚ 1424.35546875 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda.out‚ 216.7734375" disk.nvme0c0n1.in‚ 601.26953125 proc.cpu.threads‚17 disk.dm-1.in‚0) proc.memory.percent‚17.53401756286621 cpu‚25.366516905344447«·}¡2é- _wandbruntime‚5577  _runtime‚5577Þ„»ÇêÁ & lr-AdamW/pg2‚9.058210622332248e-05 trainer/global_step‚7349 % _timestamp‚1.7884338925096083e+09 _runtime‚5577.214338896 _step‚602 & lr-AdamW/pg1‚9.058210622332248e-05N,M½ÌëÆ & lr-AdamW/pg1‚9.058210622332248e-05 & lr-AdamW/pg2‚9.058210622332248e-05 trainer/global_step‚7349 % _timestamp‚1.7884338925096083e+09 _runtime‚5577.214338896 _step‚602Ú7.Ö~2ì- _wandbruntime‚5577  _runtime‚5577¬3yä'íÞO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467470604406272 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚428133498905088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6140744239572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428133498905088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428133480902144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138895836 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107109866 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107109864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138915874 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6140089801816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35252328 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚603 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚416902084090880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6162782360576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚467470580466176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107109864 _runtime‚5577.215293425 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138895836 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62499012 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚416902101130240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚434296937628160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434296263262720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35251914 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97751342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62499014 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246005700 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6465516124160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6163438723072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424nÿôé'îãO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚416902084090880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6162782360576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚467470580466176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107109864 _runtime‚5577.215293425 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138895836 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62499012 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚416902101130240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚434296937628160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434296263262720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35251914 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97751342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62499014 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246005700 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6465516124160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6163438723072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467470604406272 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚428133498905088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6140744239572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428133498905088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428133480902144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138895836 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107109866 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107109864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138915874 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6140089801816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35252328 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚603 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908ÛCYO42ï- _wandbruntime‚5577  _runtime‚55779ÐRuªðéM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138934354 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62507180 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚467532255143936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107123880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884338931642077e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246038170 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62507182 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473998800603648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6141560674744 _runtime‚5577.869412236 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚7349 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6164258164224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246058236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35256344 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246038170 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97763524 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246058236 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97763960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚423097894458220 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107123882 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6163601788416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6466545459712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428189833940992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467532231203840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416956988221440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.alÞÖØ&El.allocated‚423098565935544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138914290 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6163601788416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107123880 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138914290 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6140906236780 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138934354 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428189851943936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428189833940992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚434354110108160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚434353435729408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚467532255143936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107123880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884338931642077e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246038170 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62507182 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473998800603648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6141560674744 _runtime‚5577.869412236 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚7349 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6164258164224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246058236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35256344 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246038170 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97763524 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246058236 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97763960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚423097894458220 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107123882 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6163601788416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6466545459712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428189833940992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467532231203840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416956988221440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚423098565935544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138914290 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6163601788416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107123880 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138914290 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6140906236780 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138934354 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428189851943936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428189833940992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚434354110108160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚434353435729408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35477400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚477061773422080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6203768166400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139821888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35476962 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425828215760104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139821888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425828887237220 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6180925141224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚7399 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚470553141289472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437154901010944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6204424528896 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437154901010944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6203768166400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247652592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚430951150847488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98384374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚477061585858048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚607 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247632552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107810664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107810664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6181579578980 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚437155575376384 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430a -€äO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247632552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107810664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107810664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6181579578980 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚437155575376384 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 62044O/½#24528896 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚437155575376384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6508468508672 % _timestamp‚1.7884339275642095e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107810666 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139841926 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚5612.269096929 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚470553117349376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚430951132844544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚98384814 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35477400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚477061773422080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6203768166400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139821888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35476962 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425828215760104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139821888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425828887237220 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6180925141224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚7399 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚470553141289472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437154901010944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6204424528896 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437154901010944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6203768166400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247652592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚430951150847488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98384374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚477061585858048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚607ß”¿“c2- _wandbruntime‚5612  _runtime‚5612BÚP©ï&‚éM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107824682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚470614792027136 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚431007503886336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35481442 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6205243970048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚437212073477632 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6204587594240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247665022 _step‚608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247685088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚477124095368704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437212747856384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247665022 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139840342 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247685088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431007485883392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884339282194705e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425884607802952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35481891 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107824682 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98397022 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6509495086592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431007503886336 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419702211788800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚419702194749440 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431007485883392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚477124287113728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470614768087040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139840342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62915582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚437212747856384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6509327281664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425883936325628 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139860406 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107824680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6204587594240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 !ô&ƒîM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107824682 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98397022 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6509495086592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431007503886336 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419702211788800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚419702194749440 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431007485883392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚477124287113728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470614768087040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139840342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62915582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚437212747856384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6509327281664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425883936325628 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139860406 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107824680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6204587594240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107824682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚470614792027136 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚431007503886336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35481442 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6205243970048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚437212073477632 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6204587594240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247665022 _step‚608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247685088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚477124095368704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437212747856384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247665022 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139840342 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247685088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431007485883392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884339282194705e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425884607802952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35481891 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928à&8¼2„- _wandbruntime‚5612  _runtime‚5612ó‡›Ë_…Ù 1loss/FBt2/assignment_loss‚0.17170735749532468 1loss/FBt4/assignment_loss‚0.23394563503097743 .loss/FRt4/detection_loss‚2.563466115770015 1loss/SRqqt2/detection_loss‚1.7747345158347376 epoch‚6 % _timestamp‚1.7884339282212849e+09 /loss/FRt2/detection_loss‚2.4248117562383413 _step‚609 0loss/SRqqt3/detection_loss‚2.364180656467996 0loss/FBt1/detection_loss‚0.35767547384974485 /loss/FBt4/detection_loss‚2.5186099484562874 /loss/FRt4/assignment_loss‚3.346242533488707 2loss/SRqqt1/assignment_loss‚0.9971908289281761 2loss/SRqqt3/assignment_loss‚1.7460549513741237 2loss/SRqqt4/assignment_loss‚1.6496138226460009 0loss/FRt1/assignment_loss‚1.3008067299212729 1loss/SRqqt1/detection_loss‚0.8136489592383013 _runtime‚5612.925890325 2loss/SRqqt2/assignment_loss‚1.4758970157364038 1loss/FBt1/assignment_loss‚0.16420161197460126 .loss/FRt1/detection_loss‚1.601974577110793 1loss/FBt3/assignment_loss‚0.27237181677871075 /loss/FRt3/detection_loss‚3.0542935153533675 1loss/SRqqt4/detection_loss‚1.8153792847605312 .loss/FBt2/detection_loss‚2.733193665742874 /loss/FBt3/detection_loss‚2.5627243717511496 %loss/total_loss‚42.16098198421241 trainer/global_step‚7399 0loss/FRt2/assignment_loss‚3.1222962334752085 0loss/FRt3/assignment_loss‚3.0959606067887666·a$½d†Þ 2loss/SRqqt3/assignment_loss‚1.7460549513741237 2loss/SRqqt4/assignment_loss‚1.6496138226460009 0loss/FRt1/assignment_loss‚1.3008067299212729 1loss/SRqqt1/detection_loss‚0.8136489592383013 _runtime‚5612.925890325 2loss/SRqqt2/assignment_loss‚1.4758970157364038 1loss/FBt1/assignment_loss‚0.16420161197460126 .loss/FRt1/detection_loss‚1.601974577110793 1loss/FBt3/assignment_loss‚0.27237181677871075 /loss/FRt3/detection_loss‚3.0542935153533675 1loss/SRqqt4/detection_loss‚1.8153792847605312 .loss/FBt2/detection_loss‚2.733193665742874 /loss/FBt3/detection_loss‚2.5627243717511496 %loss/total_loss‚42.16098198421241 trainer/global_step‚7399 0loss/FRt2/assignment_loss‚3.1222962334752085 0loss/FRt3/assignment_loss‚3.0959606067887666 1loss/FBt2/assignment_loss‚0.17170735749532468 1loss/FBt4/assignment_loss‚0.23394563503097743 .loss/FRt4/detection_loss‚2.563466115770015 1loss/SRqqt2/detection_loss‚1.7747345158347376 epoch‚6 % _timestamp‚1.7884339282212849e+09 /loss/FRt2/detection_loss‚2.4248117562383413 _step‚609 0loss/SRqqt3/detection_loss‚2.364180656467996 0loss/FBt1/detection_loss‚0.35767547384974485 /loss/FBt4/detection_loss‚2.5186099484562874 /loss/FRt4/assignment_loss‚3.346242533488707 2loss/SRqqt1/assignment_loss‚0.9971908289281761áQ,Û>ž‡‚(:“ ”¬åÔ‹±¦Ž gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.136' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.027199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚“Â\ˆ‚(:¦ ”¬åÔœ™²‘" proc.memory.rssMB‚ 2872.7734375 disk.sda3.out‚ 219.26953125 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 network.sent‚ 1.7222937e+07 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 601.26953125 disk.sda1.in‚0 µkðÑÕdisk.nvme0n1.in‚ 601.26953125 network.recv‚ 1.967736e+06! disk.rbd2.out‚159.47607421875 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0) proc.memory.percent‚17.53401756286621 cpu‚25.363100663590863 proc.cpu.threads‚17) disk./.usagePercent‚73.87590189406252" disk.nvme0c0n1.in‚ 601.26953125 disk.sda2.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚33532.7607421875% disk./.usageGB‚1100.8043251037598 disk.dm-1.out‚ 219.26953125$ memory_percent‚91.30926132202148' proc.memory.availableMB‚ 1423.890625' disk.nvme0c0n1.out‚33532.7607421875 disk.rbd0.out‚0" disk.dm-0.out‚33532.7607421875 disk.sda.out‚ 219.26953125 disk.rbd1.out‚ 3.00341796875Õ›“‰‚(:ˆ £¬åÔÚ€ø gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.292' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.4584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ»Y.Š‚(:£ £¬åÔÕøü( disk./.usagePercent‚73.8758762933091 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 219.9140625 disk.sda2.out‚0 disk.sda3.out‚ 219.9140625) proc.memory.percent‚17.53401756286621 proc.cpu.threads‚17 disk.rbd2.in‚0! disk.rbd2.out‚161.00341796875 network.sent‚ 1.7331122e+07 disk.rbd3.in‚0# disk.dm-0.out‚33632.11865234375 disk.sda.out‚ 219.9140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚91.30635261535645 disk.rbd3.out‚0 disk.nvme0n1.in‚ 601.26953125% disk./.usageGB‚1100.8039436340332 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.06640625 disk.sda2.in‚0 disk.sda3.in‚0 cpu‚25.362284892799938 disk.dm-0.in‚ 601.26953125& disk.nvme0n1.out‚33632.11865234375( proc.memory.availableMB‚ 1424.3671875" proc.memory.rssMB‚ 2872.7734375 network.recv‚ 1.974287e+06 disk.sda.in‚0( disk.nvme0c0n1.out‚33632.11865234375 disk.rbd0.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 601.26953125ݬ2‹- _wandbruntime‚5647  _runtime‚5647˜æ(ÇŒÁ & lr-AdamW/pg1‚8.738696055756626e-05 & lr-AdamW/pg2‚8.738696055756626e-05 trainer/global_step‚7449 % _timestamp‚1.7884339624709642e+09 _runtime‚5647.175701791 _step‚6103ø…ÌÆ & lr-AdamW/pg1‚8.738696055756626e-05 & lr-AdamW/pg2‚8.738696055756626e-05 trainer/global_step‚7449 % _timestamp‚1.7884339624709642e+09 _runtime‚5647.175701791 _step‚610âDªs2Ž- _wandbruntime‚5647  _runtime‚5647Ì0zmä'ÞO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108511466 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433768802789888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221760480632 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433768802789888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249279444 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140747940 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6244753972224 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108511464 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚422392514186240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚440014213124608 _runtime‚5647.176614581 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249259404 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚440013538759168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473635678172672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140767978 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473635654232576 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6244753972224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚420 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480186992055808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 $ _timestamp‚1.788433962471675e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140767978 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99018649 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108511466 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6222414918388 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6245410334720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6245410334720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140747940 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚428614257627512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6551337823232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440013538759168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63315814 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚440014213124608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚611 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433768784786944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63315812 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚422392497146880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6244753972224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚420 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480186992055808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 $ _timestamp‚1.788433962471675e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140767978 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99018649 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108511466 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6222414918388 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6245410334720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6245410334720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140747940 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚428614257627512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6551337823232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440013538759168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63315814 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚440014213124608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚611 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433768784786944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63315812 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚422392497146880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221760480632 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433768802789888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249279444 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140747940 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6244753972224 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108511464 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚422392514186240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚440014213124608 _runtime‚5647.176614581 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249259404 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚440013538759168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473635678172672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140767978 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2ã4˜P2‘- _wandbruntime‚5647  _runtime‚5647œÄ¤'%’éM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚440070711225856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473697328910336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108525482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480249500493312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚428669978193036 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6552195523072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6246229775872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99031252 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433825155828736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249311940 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884339631251075e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63323980 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6222576915596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚422447401277440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140786458 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433825137825792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6245573400064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433825137825792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63323982 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6246229775872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108525482 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚7449 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5647.830656611 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108525480 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚433825155828736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚428670649670360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35707711 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249291874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚422447418316800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚ѺñØÈ844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140766394 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249291874 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249311940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35707272è¶Mô&“îM EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99031252 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433825155828736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249311940 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884339631251075e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63323980 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6222576915596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚422447401277440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140786458 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433825137825792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6245573400064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433825137825792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63323982 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6246229775872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108525482 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚7449 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5647.830656611 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108525480 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚433825155828736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚428670649670360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35707711 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249291874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚422447418316800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140766394 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249291874 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249311940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35707272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚440070711225856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473697328910336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108525482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480249500493312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚428669978193036 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6552195523072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6246229775872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ä…­32”- _wandbruntime‚5647  _runtime‚5647õjx[•Õ /loss/FBt1/detection_loss‚0.7553469007375689 _runtime‚ 5647.83184993 _step‚613 1loss/FBt1/assignment_loss‚0.06941892948119557 2loss/FBt2/assignment_loss‚0.011165946251518713 /loss/FRt1/assignment_loss‚1.192517933980474 1loss/SRqqt3/assignment_loss‚1.600016457278554 2loss/SRqqt4/assignment_loss‚1.2959326060116292 .loss/FRt4/detection_loss‚2.439582119137049 /loss/FRt2/detection_loss‚2.3864333268175733 /loss/FBt4/detection_loss‚2.6857670599764045 trainer/global_step‚7449 0loss/FRt2/assignment_loss‚3.1265282580193055 /loss/FRt3/assignment_loss‚3.220947515579962 0loss/FBt4/assignment_loss‚0.1643302349204367 .loss/FRt3/detection_loss‚2.876353105229716 1loss/SRqqt1/detection_loss‚0.8937569568826728 1loss/SRqqt2/detection_loss‚1.6684321837042861 %loss/total_loss‚42.32448485310701 .loss/FBt2/detection_loss‚4.245172023773193 epoch‚6 % _timestamp‚1.7884339631271632e+09 0loss/FRt4/assignment_loss‚2.9829413130879403 2loss/SRqqt2/assignment_loss‚1.6701857286839272 1loss/SRqqt3/detection_loss‚1.9767726819689682 1loss/FBt3/assignment_loss‚0.04950182663742453 2loss/SRqqt1/assignment_loss‚0.8508639933082135 1loss/SRqqt4/detection_loss‚1.7341933864355088 .loss/FBt3/detection_loss‚2.776676813761393 /loss/FRt1/detection_loss‚1.6516475514420923GË¢`–Ú 0loss/FBt4/assignment_loss‚0.1643302349204367 .loss/FRt3/detection_loss‚2.876353105229716 1loss/SRqqt1/detection_loss‚0.8937569568826728 1loss/SRqqt2/detection_loss‚1.6684321837042861 %loss/total_loss‚42.32448485310701 .loss/FBt2/detection_loss‚4.245172023773193 epoch‚6 % _timestamp‚1.7884339631271632e+09 0loss/FRt4/assignment_loss‚2.9829413130879403 2loss/SRqqt2/assignment_loss‚1.6701857286839272 1loss/SRqqt3/detection_loss‚1.9767726819689682 1loss/FBt3/assignment_loss‚0.04950182663742453 2loss/SRqqt1/assignment_loss‚0.8508639933082135 1loss/SRqqt4/detection_loss‚1.7341933864355088 .loss/FBt3/detection_loss‚2.776676813761393 /loss/FRt1/detection_loss‚1.6516475514420923 /loss/FBt1/detection_loss‚0.7553469007375689 _runtime‚ 5647.83184993 _step‚613 1loss/FBt1/assignment_loss‚0.06941892948119557 2loss/FBt2/assignment_loss‚0.011165946251518713 /loss/FRt1/assignment_loss‚1.192517933980474 1loss/SRqqt3/assignment_loss‚1.600016457278554 2loss/SRqqt4/assignment_loss‚1.2959326060116292 .loss/FRt4/detection_loss‚2.439582119137049 /loss/FRt2/detection_loss‚2.3864333268175733 /loss/FBt4/detection_loss‚2.6857670599764045 trainer/global_step‚7449 0loss/FRt2/assignment_loss‚3.1265282580193055 /loss/FRt3/assignment_loss‚3.220947515579962å ·ò1“—‚(:ˆ ²¬åÔÊ”ù gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚130.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß“3*˜‚(:Ÿ ²¬åÔЦ­' proc.memory.availableMB‚ 1424.359375" proc.memory.rssMB‚ 2872.7734375 disk.rbd0.in‚0 disk.sda.out‚ 222.140625 disk.sda1.out‚0 disk.dm-1.out‚ 222.140625% disk./.usageGB‚1100.8039474487305 disk.rbd0.out‚0 disk.sda3.out‚ 222.140625 disk.rbd3.out‚0 disk.sda.in‚0) disk./.usagePercent‚73.87587654931663# disk.dm-0.out‚33738.27392578125" disk.nvme0c0n1.in‚ 601.26953125$ memory_percent‚91.30640029907227 disk.dm-0.in‚ 601.26953125 disk.dm-1.in‚0 cpu‚25.36141280540753 network.recv‚ 1.977284e+06 disk.rbd3.in‚0& disk.nvme0n1.out‚33738.27392578125 disk.sda2.in‚0! disk.rbd2.out‚161.06591796875 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 601.26953125) proc.memory.percent‚17.53401756286621 proc.cpu.threads‚17 network.sent‚ 1.7335517e+07 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.06640625( disk.nvme0c0n1.out‚33738.27392578125 disk.sda2.out‚0 disk.drbd1777.in‚0“Fòž™‚(:“ Á¬åÔêæ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.471' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.294199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðò*š‚(:Ÿ Á¬åÔ‚¼Í# disk.dm-0.out‚33837.32861328125 disk.sda.in‚0 disk.dm-1.in‚0) proc.memory.availableMB‚ 1424.46484375) disk./.usagePercent‚73.87588038942965 disk.rbd1.out‚ 3.06640625% disk./.usageGB‚1100.8040046691895 disk.sda1.in‚0$ memory_percent‚91.30575656890869 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 601.26953125( disk.nvme0c0n1.out‚33837.32861328125 disk.rbd0.out‚0 disk.dm-0.in‚ 601.26953125& disk.nvme0n1.out‚33837.32861328125 network.sent‚ 1.7443345e+07 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2872.7734375! disk.rbd2.out‚161.08154296875 disk.sda.out‚ 222.90625 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 222.90625) proc.memory.percent‚17.53401756286621 cpu‚25.360590401820307 disk.rbd0.in‚0 disk.sda3.out‚ 222.90625 disk.nvme0n1.in‚ 601.26953125 network.recv‚ 1.982468e+06 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd3.in‚01gàô2›- _wandbruntime‚5682  _runtime‚5682q>ÜÅœ¿ & lr-AdamW/pg1‚8.579732544548692e-05 & lr-AdamW/pg2‚8.579732544548692e-05 trainer/global_step‚7499 # _timestamp‚1.78843399768762e+09 _runtime‚5682.392346573 _step‚614$=fÌÊÄ _step‚614 & lr-AdamW/pg1‚8.579732544548692e-05 & lr-AdamW/pg2‚8.579732544548692e-05 trainer/global_step‚7499 # _timestamp‚1.78843399768762e+09 _runtime‚5682.392346573æ­þK+2ž- _wandbruntime‚5682  _runtime‚5682)ž.äå'ŸßO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141673992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚483312681038848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6263250257796 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431400970972036 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6262595820040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442872176507392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35928330 _step‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚442872850872832 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚442872850872832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6285739778048 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚5682.393230659 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35928776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436586436729344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109212266 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63724212 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6286396140544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436586454732288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚436586454732288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 trainer/global_step‚7499 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚442872176507392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 % _timestamp‚1.7884339976883307e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038i©‘ äO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109212266 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63724212 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6286396140544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436586454732288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚436586454732288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 trainer/global_step‚7499 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚442872176507392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 % _timestamp‚1.7884339976883307e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141673992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚483312681038848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6263250257796 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431400970972036 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6262595820040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442872176507392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35928330 _step‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚442872850872832 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚442872850872832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6285739778048 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚5682.393230659 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35928776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436586436729344ç^;ð2¡- _wandbruntime‚5683  _runtime‚5683Ó| 8î&¢èM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚7499 _runtime‚5683.078006911 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431456020060444 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚442930023352832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425192624844800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442929348974080 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚436642789768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63732382 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6595321332736 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚483374995381248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99665657 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99665217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250938792 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141712510 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141712510 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚476779865793536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6264066692968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6287215581696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431456691537768 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35933275 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6287215581696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442930023352832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109226282 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250918726 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425192607805440 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109226280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 $ _timestamp‚1.788433998372849e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109226280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250938792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6263412255004 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚436642807771136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63732380 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚483375187126272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6595153527808 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141692446ÿb?ó&£íM F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6287215581696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442930023352832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109226282 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250918726 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425192607805440 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109226280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 $ _timestamp‚1.788433998372849e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109226280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250938792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6263412255004 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚436642807771136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63732380 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚483375187126272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6595153527808 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141692446 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚7499 _runtime‚5683.078006911 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431456020060444 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚442930023352832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425192624844800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442929348974080 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚436642789768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63732382 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6595321332736 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚483374995381248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99665657 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99665217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250938792 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141712510 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141712510 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚476779865793536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6264066692968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6287215581696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431456691537768 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35933275 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0袼/2¤- _wandbruntime‚5683  _runtime‚5683M:È›[¥Õ 0loss/FRt1/assignment_loss‚1.3725557573463605 /loss/FRt2/assignment_loss‚2.654151043721608 1loss/FBt2/assignment_loss‚0.18115204319999773 1loss/SRqqt3/detection_loss‚2.5333816916854293 %loss/total_loss‚41.53369006339791 trainer/global_step‚7499 2loss/SRqqt1/assignment_loss‚0.9014446216768452 1loss/FBt4/assignment_loss‚0.19826336698606611 .loss/FRt4/detection_loss‚2.257238975912412 1loss/SRqqt2/detection_loss‚1.6339966266282968 1loss/SRqqt4/detection_loss‚2.2169213008135555 .loss/FBt1/detection_loss‚0.542554909328091 /loss/FBt2/detection_loss‚1.6510686013433669 epoch‚6 /loss/FRt3/assignment_loss‚3.618805155158043 2loss/SRqqt2/assignment_loss‚1.2589500101416238 1loss/FBt1/assignment_loss‚0.12011428427811198 /loss/FRt1/detection_loss‚1.6300322837803676 .loss/FRt3/detection_loss‚2.715283067524433 % _timestamp‚1.7884339983745186e+09 _runtime‚5683.079143883 1loss/FBt3/assignment_loss‚0.08900804865233011 1loss/SRqqt1/detection_loss‚1.0006631785737616 .loss/FBt4/detection_loss‚3.867288959026337 _step‚617 /loss/FRt2/detection_loss‚2.2897283420676278 /loss/FRt4/assignment_loss‚3.066811215877533 /loss/FBt3/detection_loss‚2.4342259301079645 1loss/SRqqt3/assignment_loss‚1.747248257751818 2loss/SRqqt4/assignment_loss‚1.5528023918159306. e`¦Ú /loss/FRt4/assignment_loss‚3.066811215877533 /loss/FBt3/detection_loss‚2.4342259301079645 1loss/SRqqt3/assignment_loss‚1.747248257751818 2loss/SRqqt4/assignment_loss‚1.5528023918159306 0loss/FRt1/assignment_loss‚1.3725557573463605 /loss/FRt2/assignment_loss‚2.654151043721608 1loss/FBt2/assignment_loss‚0.18115204319999773 1loss/SRqqt3/detection_loss‚2.5333816916854293 %loss/total_loss‚41.53369006339791 trainer/global_step‚7499 2loss/SRqqt1/assignment_loss‚0.9014446216768452 1loss/FBt4/assignment_loss‚0.19826336698606611 .loss/FRt4/detection_loss‚2.257238975912412 1loss/SRqqt2/detection_loss‚1.6339966266282968 1loss/SRqqt4/detection_loss‚2.2169213008135555 .loss/FBt1/detection_loss‚0.542554909328091 /loss/FBt2/detection_loss‚1.6510686013433669 epoch‚6 /loss/FRt3/assignment_loss‚3.618805155158043 2loss/SRqqt2/assignment_loss‚1.2589500101416238 1loss/FBt1/assignment_loss‚0.12011428427811198 /loss/FRt1/detection_loss‚1.6300322837803676 .loss/FRt3/detection_loss‚2.715283067524433 % _timestamp‚1.7884339983745186e+09 _runtime‚5683.079143883 1loss/FBt3/assignment_loss‚0.08900804865233011 1loss/SRqqt1/detection_loss‚1.0006631785737616 .loss/FBt4/detection_loss‚3.867288959026337 _step‚617 /loss/FRt2/detection_loss‚2.2897283420676278éH¾Ûž§‚(:“ ЬåÔî‰Ý gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.003' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.800599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03G´¬.¨‚(:£ ЬåÔ…¦ƒ network.sent‚ 1.7447079e+07 disk.sda.in‚0 disk.sda.out‚ 223.25390625 disk.sda1.in‚0 disk.dm-1.in‚0 network.recv‚ 1.985491e+06 disk.rbd2.out‚ 161.23046875 disk.sda3.out‚ 223.25390625 disk.drbd1777.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 1424.4765625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 601.26953125& disk.nvme0n1.out‚33837.62939453125% disk./.usageGB‚1100.8040046691895 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 2872.7734375 disk.rbd0.in‚0# disk.dm-0.out‚33837.62939453125 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 601.26953125) proc.memory.percent‚17.53401756286621 cpu‚25.35973620890106 disk.sda3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚33837.62939453125 disk.sda2.out‚0 disk.nvme0n1.in‚ 601.26953125 disk.rbd1.out‚ 3.06640625$ memory_percent‚91.30568504333496) disk./.usagePercent‚73.87588038942965 disk.dm-1.out‚ 223.25390625‰&ä“©‚(:ˆ ߬åÔÁØå gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.083' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ÛÛä*ª‚(:Ÿ ߬åÔØï§% disk./.usageGB‚1100.8039817810059 disk.sda.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚33936.78955078125 disk.drbd1777.out‚0 disk.rbd3.out‚0 cpu‚25.358965849793133" disk.nvme0c0n1.in‚ 601.26953125 disk.sda2.in‚0 disk.rbd0.out‚0$ memory_percent‚94.20089721679688# proc.memory.availableMB‚950.125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd1.in‚0) proc.memory.percent‚17.53401756286621 network.sent‚ 1.7555147e+07 disk.rbd2.in‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 2872.7734375# disk.dm-0.out‚33936.78955078125 disk.sda.out‚ 223.86328125 disk.nvme0n1.in‚ 601.26953125 disk.rbd0.in‚0) disk./.usagePercent‚73.87587885338445 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.recv‚ 1.991459e+06 disk.dm-0.in‚ 601.26953125 disk.rbd2.out‚ 161.24609375 disk.drbd1777.in‚0 disk.dm-1.out‚ 223.86328125& disk.nvme0n1.out‚33936.78955078125 disk.rbd1.out‚ 3.06640625 disk.sda3.out‚ 223.86328125‰­“«‚(:ˆ î¬åÔ¢£ªŽ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚130.548' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.1096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÆ.¬‚(:£ î¬åÔÝçÑ( disk.nvme0c0n1.out‚34036.70751953125 disk.sda2.in‚0 disk.sda3.out‚ 224.19921875 disk.drbd1777.in‚0 disk.sda.out‚ 224.19921875 disk.nvme0n1.in‚ 601.26953125 disk.rbd1.in‚0' proc.memory.availableMB‚ 950.3671875 proc.cpu.threads‚17 disk.sda1.out‚0 network.sent‚ 1.7559308e+07 network.recv‚ 1.993934e+06) disk./.usagePercent‚73.87587885338445 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 601.26953125" proc.memory.rssMB‚ 2872.7734375) proc.memory.percent‚17.53401756286621 disk.rbd2.out‚ 161.30859375 disk.dm-0.in‚ 601.26953125 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 3.06640625$ memory_percent‚94.19941902160645% disk./.usageGB‚1100.8039817810059 disk.rbd0.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚34036.70751953125& disk.nvme0n1.out‚34036.70751953125 cpu‚25.358112025800903 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 224.19921875Ùðƒ52­- _wandbruntime‚5717  _runtime‚5717 ëÎÙÇ®Á & lr-AdamW/pg2‚8.421342422600535e-05 trainer/global_step‚7549 % _timestamp‚1.7884340326742852e+09 _runtime‚5717.379034711 _step‚618 & lr-AdamW/pg1‚8.421342422600535e-05-.L&Ì¯Æ trainer/global_step‚7549 % _timestamp‚1.7884340326742852e+09 _runtime‚5717.379034711 _step‚618 & lr-AdamW/pg1‚8.421342422600535e-05 & lr-AdamW/pg2‚8.421342422600535e-05ê&ØpÓ2°- _wandbruntime‚5717  _runtime‚5717°…è±àO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 H>DeviceStatsMonitor.onÿX¦þ%_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚434187012839444 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚427882910202880 _runtime‚5717.379975734 trainer/global_step‚7549 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚486438065331712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚486437879864832 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100286789 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36154175 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6637313392640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚434186341362328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 $ _timestamp‚1.788434032675038e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252513108 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚439404088671744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445730814255616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6327381946368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439404088671744 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚619 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142600044 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚439404106674688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252533148 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142600044 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142620082 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427882927242240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6303431159448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445731488621056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64132612 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6327381946368 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100286365 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64132614 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445731488621056 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439404106674688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚479800751939072 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109913066 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109913064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚479800727998976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6326725583872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109913064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3375385605{L‹ë'²åO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142600044 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚439404106674688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252533148 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142600044 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142620082 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427882927242240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6303431159448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445731488621056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64132612 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6327381946368 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100286365 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64132614 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445731488621056 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439404106674688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚479800751939072 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109913066 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109913064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚479800727998976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6326725583872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109913064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚434187012839444 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚427882910202880 _runtime‚5717.379975734 trainer/global_step‚7549 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚486438065331712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚486437879864832 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100286789 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36154175 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6637313392640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚434186341362328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 $ _timestamp‚1.788434032675038e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252513108 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚439404088671744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445730814255616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6327381946368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439404088671744 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚619ëâÂÉu2³- _wandbruntime‚5718  _runtime‚5718—â¥ð&´êM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚421 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427937814333440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6328201387520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚445787986722304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252565644 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252565644 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439460459713536 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6328201387520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚7549 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 $ _timestamp‚1.788434033333632e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64140782 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚5718.038884936 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109927082 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100299446 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439460441710592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6638182973952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚479862378736640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109927080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252545578 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142618498 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142638562 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445788661101056 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚486500406488576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142638562 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109927082 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36158664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252545578 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6327545011712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109927080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚419 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142618498 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100299025 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928ߦŽÒÒ µïM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 $ _timestamp‚1.788434033333632e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64140782 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚5718.038884936 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109927082 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100299446 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439460441710592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6638182973952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚479862378736640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109927080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252545578 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142618498 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current½'qv#‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142638562 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445788661101056 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚486500406488576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142638562 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109927082 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36158664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252545578 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6327545011712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109927080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚419 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142618498 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100299025 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚421 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427937814333440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6328201387520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚445787986722304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252565644 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252565644 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439460459713536 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6328201387520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚7549ì~[bª2¶- _wandbruntime‚5718  _runtime‚5718¿Þ%f]·× 0loss/SRqqt1/detection_loss‚1.148331546331527 /loss/FBt1/detection_loss‚0.5653291760184772 .loss/FBt4/detection_loss‚2.584452490011851 1loss/SRqqt2/assignment_loss‚1.599643625319004 2loss/SRqqt4/assignment_loss‚1.3396623278812816 2loss/FBt2/assignment_loss‚0.058921385256326175 _runtime‚5718.039958037 0loss/FBt1/assignment_loss‚0.1986154698967381 &loss/total_loss‚42.707230701642374 trainer/global_step‚7549 /loss/FRt1/detection_loss‚2.0057815324677075 /loss/FRt2/detection_loss‚2.3262789520834173 /loss/FBt3/detection_loss‚2.7288790430341447 % _timestamp‚1.7884340333353539e+09 0loss/FRt4/assignment_loss‚3.1589107667958296 0loss/FRt1/assignment_loss‚1.4140460776246113 /loss/FRt2/assignment_loss‚3.243769059223788 /loss/FRt3/assignment_loss‚3.600495599565052 2loss/SRqqt3/assignment_loss‚1.7516125438114007 0loss/FBt3/assignment_loss‚0.1893878799497283 1loss/SRqqt2/detection_loss‚1.7940538081587578 0loss/SRqqt4/detection_loss‚1.669206861158212 2loss/SRqqt1/assignment_loss‚1.0443032000702424 /loss/FRt3/detection_loss‚3.1763944767770314 /loss/FBt2/detection_loss‚2.5202723294496536 epoch‚6 _step‚621 1loss/SRqqt3/detection_loss‚2.4950279076894124 1loss/FBt4/assignment_loss‚0.12013566129220028 /loss/FRt4/detection_loss‚1.9737189817759884÷ƪ b¸Ü /loss/FRt1/detection_loss‚2.0057815324677075 /loss/FRt2/detection_loss‚2.3262789520834173 /loss/FBt3/detection_loss‚2.7288790430341447 % _timestamp‚1.7884340333353539e+09 0loss/FRt4/assignment_loss‚3.1589107667958296 0loss/FRt1/assignment_loss‚1.4140460776246113 /loss/FRt2/assignment_loss‚3.243769059223788 /loss/FRt3/assignment_loss‚3.600495599565052 2loss/SRqqt3/assignment_loss‚1.7516125438114007 0loss/FBt3/assignment_loss‚0.1893878799497283 1loss/SRqqt2/detection_loss‚1.7940538081587578 0loss/SRqqt4/detection_loss‚1.669206861158212 2loss/SRqqt1/assignment_loss‚1.0443032000702424 /loss/FRt3/detection_loss‚3.1763944767770314 /loss/FBt2/detection_loss‚2.5202723294496536 epoch‚6 _step‚621 1loss/SRqqt3/detection_loss‚2.4950279076894124 1loss/FBt4/assignment_loss‚0.12013566129220028 /loss/FRt4/detection_loss‚1.9737189817759884 0loss/SRqqt1/detection_loss‚1.148331546331527 /loss/FBt1/detection_loss‚0.5653291760184772 .loss/FBt4/detection_loss‚2.584452490011851 1loss/SRqqt2/assignment_loss‚1.599643625319004 2loss/SRqqt4/assignment_loss‚1.3396623278812816 2loss/FBt2/assignment_loss‚0.058921385256326175 _runtime‚5718.039958037 0loss/FBt1/assignment_loss‚0.1986154698967381 &loss/total_loss‚42.707230701642374 trainer/global_step‚7549íãƒ%¹‚(:’ ý¬åÔ¸ƒ…Ž gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.566000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Íî{$º‚(:™ ý¬åÔü·Š’" disk.nvme0c0n1.in‚ 601.26953125) proc.memory.percent‚17.53401756286621 cpu‚25.35729596400425 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0$ memory_percent‚94.20311450958252 network.sent‚ 1.7563831e+07 disk.dm-0.in‚ 601.26953125# disk.dm-0.out‚34135.85205078125 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 1.996383e+06 disk.rbd0.in‚0 disk.sda.out‚ 224.96875 disk.dm-1.out‚ 224.96875 disk.nvme0n1.in‚ 601.26953125& disk.nvme0n1.out‚34135.85205078125 disk.rbd1.out‚ 3.06640625) disk./.usagePercent‚73.87587885338445" proc.memory.rssMB‚ 2872.7734375( disk.nvme0c0n1.out‚34135.85205078125 disk.rbd2.out‚ 162.8203125 disk.drbd1777.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 949.76171875 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0% disk./.usageGB‚1100.8039817810059 disk.sda3.out‚ 224.96875 disk.drbd1777.in‚0°Pé ž»‚(:“ Œ­åÔ²¡þ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚105.401' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.080199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÀs$¼‚(:™ Œ­åÔ´ÅÀ disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd2.in‚0) proc.memory.percent‚17.53401756286621 cpu‚25.356511569820892 proc.cpu.threads‚17% disk./.usageGB‚1100.8039817810059" disk.nvme0c0n1.in‚ 601.26953125' proc.memory.availableMB‚ 949.9765625 disk.drbd1777.out‚0 disk.sda.out‚ 225.69921875" proc.memory.rssMB‚ 2872.7734375) disk./.usagePercent‚73.87587885338445 disk.rbd3.out‚0 disk.dm-0.in‚ 601.26953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 225.69921875% disk.nvme0c0n1.out‚34235.04296875 disk.dm-0.out‚34235.04296875 disk.nvme0n1.in‚ 601.26953125 disk.sda2.out‚0 disk.rbd2.out‚ 162.9609375$ memory_percent‚94.20180320739746 network.recv‚ 2.001216e+06 disk.sda3.in‚0 disk.sda3.out‚ 225.69921875 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 network.sent‚ 1.7670614e+07 disk.sda.in‚0# disk.nvme0n1.out‚34235.04296875 disk.rbd1.out‚ 3.06640625 disk.rbd0.in‚0‹ÉI€2½- _wandbruntime‚5752  _runtime‚5752cØÅǾÁ trainer/global_step‚7599 % _timestamp‚1.7884340679568388e+09 _runtime‚5752.661582106 _step‚622 & lr-AdamW/pg1‚8.263558698176175e-05 & lr-AdamW/pg2‚8.263558698176175e-05!“UÌ¿Æ % _timestamp‚1.7884340679568388e+09 _runtime‚5752.661582106 _step‚622 & lr-AdamW/pg1‚8.263558698176175e-05 & lr-AdamW/pg2‚8.263558698176175e-05 trainer/global_step‚7599îø$؃2À- _wandbruntime‚5752  _runtime‚5752•Ãç'ÁáO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143546134 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143526096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚436973054706852 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110613864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100921527 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110613866 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448589452003840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254160000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442221758617088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64541014 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚430628133770240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚5752.662474788 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110613864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6367711389696 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100921964 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64541012 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6680277188096 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143546134 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482883288822272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442221758617088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36380515 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254139960 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚448590126369280 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6368367752192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6680442909184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489563731731456 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884340679575744e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6368367752192 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448589452003840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442221740614144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6367711389696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143526096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489563542070272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚623 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚448590126369280 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110613866 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36380950 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6344920936612 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442221740614144 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482883264882176 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489563542070272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚623 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚448590126369280 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110613866 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36380950 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6344920936612 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442221740614144 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482883264882176 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚436973054706852 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110613864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100921527 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110613866 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448589452003840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254160000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442221758617088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64541014 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚430628133770240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚5752.662474788 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110613864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6367711389696 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100921964 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64541012 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6680277188096 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143546134 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482883288822272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442221758617088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36380515 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254139960 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚448590126369280 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6368367752192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6680442909184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489563731731456 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884340679575744e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMóãÜÜÃonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6368367752192 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448589452003840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442221740614144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6367711389696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ï\X!Û2Ã- _wandbruntime‚5753  _runtime‚57533o±Ñï&ÄéM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489626085638144 % _timestamp‚1.7884340686400769e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6681170018304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚430683020861440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168676352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482944939559936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6681314754560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448646624470528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110627880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437028775272584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚ 5753.3451528 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6368530817536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36385457 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482944915619840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143564614 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254172430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143564614 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64549182 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254172430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100934639 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100934211 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6345737371784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6368530817536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110627882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442278093652992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489626254314496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚430683037900800 _step‚624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254192496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254192496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 144736256 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚442278093652992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110627880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143544550 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442278111655936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚442278111655936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6369187193344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0ú_©ô&ÅîM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚442278111655936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6369187193344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489626085638144 % _timestamp‚1.7884340686400769e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6681170018304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚430683020861440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168676352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482944939559936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6681314754560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448646624470528 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110627880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437028775272584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚ 5753.3451528 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6368530817536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36385457 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482944915619840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143564614 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254172430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143564614 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64549182 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254172430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100934639 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100934211 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6345737371784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6368530817536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110627882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442278093652992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489626254314496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚430683037900800 _step‚624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254192496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254192496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 144736256 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚442278093652992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110627880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143544550 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442278111655936ðÀÁŠ2Æ- _wandbruntime‚5753  _runtime‚5753úäº^ÇØ /loss/FRt3/detection_loss‚3.0666350810616105 /loss/FRt4/detection_loss‚2.1534490152901293 epoch‚6 .loss/FBt3/detection_loss‚3.048628866672516 0loss/FRt1/assignment_loss‚1.2668540701270103 1loss/FBt1/assignment_loss‚0.11614055319343777 /loss/FRt1/detection_loss‚1.7442418929189443 1loss/SRqqt3/detection_loss‚2.4066357739043958 /loss/FBt2/detection_loss‚2.4123565091027155 %loss/total_loss‚41.13400713417073 % _timestamp‚1.7884340686415915e+09 2loss/SRqqt2/assignment_loss‚1.3659012942607034 trainer/global_step‚7599 _step‚625 2loss/SRqqt4/assignment_loss‚1.4016192976671915 1loss/FBt3/assignment_loss‚0.14259747633441294 0loss/FRt4/assignment_loss‚2.9885035028644635 2loss/SRqqt1/assignment_loss‚0.8758139980071392 2loss/SRqqt3/assignment_loss‚2.2411301029902515 0loss/SRqqt1/detection_loss‚0.964936433970684 0loss/SRqqt2/detection_loss‚1.639819807019727 .loss/FBt1/detection_loss‚0.613209140379714 /loss/FBt4/detection_loss‚2.4836193431507456 /loss/FRt2/assignment_loss‚2.894624482308115 1loss/FBt2/assignment_loss‚0.08352744959605236 1loss/FBt4/assignment_loss‚0.07617405912076886 1loss/SRqqt4/detection_loss‚1.8486114292620466 _runtime‚5753.346184134 /loss/FRt2/detection_loss‚1.9918291116399425 0loss/FRt3/assignment_loss‚3.3071484433280096'¦¨cÈÝ epoch‚6 .loss/FBt3/detection_loss‚3.048628866672516 0loss/FRt1/assignment_loss‚1.2668540701270103 1loss/FBt1/assignment_loss‚0.11614055319343777 /loss/FRt1/detection_loss‚1.7442418929189443 1loss/SRqqt3/detection_loss‚2.4066357739043958 /loss/FBt2/detection_loss‚2.4123565091027155 %loss/total_loss‚41.13400713417073 % _timestamp‚1.7884340686415915e+09 2loss/SRqqt2/assignment_loss‚1.3659012942607034 trainer/global_step‚7599 _step‚625 2loss/SRqqt4/assignment_loss‚1.4016192976671915 1loss/FBt3/assignment_loss‚0.14259747633441294 0loss/FRt4/assignment_loss‚2.9885035028644635 2loss/SRqqt1/assignment_loss‚0.8758139980071392 2loss/SRqqt3/assignment_loss‚2.2411301029902515 0loss/SRqqt1/detection_loss‚0.964936433970684 0loss/SRqqt2/detection_loss‚1.639819807019727 .loss/FBt1/detection_loss‚0.613209140379714 /loss/FBt4/detection_loss‚2.4836193431507456 /loss/FRt2/assignment_loss‚2.894624482308115 1loss/FBt2/assignment_loss‚0.08352744959605236 1loss/FBt4/assignment_loss‚0.07617405912076886 1loss/SRqqt4/detection_loss‚1.8486114292620466 _runtime‚5753.346184134 /loss/FRt2/detection_loss‚1.9918291116399425 0loss/FRt3/assignment_loss‚3.3071484433280096 /loss/FRt3/detection_loss‚3.0666350810616105 /loss/FRt4/detection_loss‚2.1534490152901293ñhö’É‚(:‡ ›­åÔøëÖ gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚96.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.3608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞÕyü.Ê‚(:£ ›­åÔÅÚ–‘ disk.rbd3.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 226.51953125& disk.nvme0n1.out‚34235.38232421875 cpu‚25.355667697079042 disk.rbd2.out‚ 162.9609375 network.recv‚ 2.005103e+06) disk./.usagePercent‚73.87588038942965 disk.rbd3.in‚0# disk.dm-0.out‚34235.38232421875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 601.26953125( disk.nvme0c0n1.out‚34235.38232421875 disk.drbd1777.out‚0( proc.memory.availableMB‚ 949.94921875 network.sent‚ 1.7676726e+07 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0% disk./.usageGB‚1100.8040046691895 disk.rbd1.in‚0 disk.sda2.out‚0$ memory_percent‚94.20197010040283" proc.memory.rssMB‚ 2872.7734375) proc.memory.percent‚17.53401756286621 proc.cpu.threads‚17 disk.dm-0.in‚ 601.26953125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 226.51953125 disk.drbd1777.in‚0 disk.sda.out‚ 226.51953125 disk.rbd1.out‚ 3.06640625" disk.nvme0c0n1.in‚ 601.26953125 disk.rbd0.in‚0—fÊ“Ë‚(:ˆ ª­åÔÛ½ð gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.337' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.8674 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y7È”1Ì‚(:¦ ª­åÔËö÷ disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.06640625$ memory_percent‚94.20197010040283& disk.nvme0n1.out‚34334.91162109375! disk.rbd2.out‚163.12158203125 disk.dm-1.out‚ 227.01171875) disk./.usagePercent‚73.87588422954266 disk.rbd3.out‚0 disk.dm-0.in‚ 601.26953125 disk.sda1.in‚0 disk.sda3.out‚ 227.01171875 cpu‚25.354888512772767% disk./.usageGB‚1100.8040618896484 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 601.26953125 disk.sda3.in‚0( disk.nvme0c0n1.out‚34334.91162109375 disk.rbd0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2872.7734375# disk.dm-0.out‚34334.91162109375 disk.drbd1777.out‚0( proc.memory.availableMB‚ 949.94921875 disk.rbd3.in‚0 disk.sda.out‚ 227.01171875 disk.nvme0n1.in‚ 601.26953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0) proc.memory.percent‚17.53401756286621 proc.cpu.threads‚17 network.sent‚ 1.7782697e+07 disk.rbd2.in‚0 disk.sda1.out‚0 network.recv‚ 2.00896e+06ô »2Í- _wandbruntime‚5787  _runtime‚57876÷ÛÜÇÎÁ & lr-AdamW/pg1‚8.106414253167282e-05 & lr-AdamW/pg2‚8.106414253167282e-05 trainer/global_step‚7649 % _timestamp‚1.7884341028122354e+09 _runtime‚5787.517009232 _step‚626ϳfÌÏÆ _runtime‚5787.517009232 _step‚626 & lr-AdamW/pg1‚8.106414253167282e-05 & lr-AdamW/pg2‚8.106414253167282e-05 trainer/global_step‚7649 % _timestamp‚1.7884341028122354e+09ò ,ø]2Ð- _wandbruntime‚5787  _runtime‚5787ɦ8ùJÑáO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144452148 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36606279 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111314666 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111314664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚5787.517904569 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚492688985939968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451448089752064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚451448089752064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445039392556544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445039410559488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6409353558016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚492689181892608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚439759096574260 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6385756276020 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445039410559488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36606722 ZLDeviceStatsMonitor.on_train_batch_start/inactive¶xU‰_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚451448764117504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚433373340298240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6723184174592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6385101838264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255786852 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144472186 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6408697195520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101556136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485965825705472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6409353558016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚7649 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101555691 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884341028129792e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111314664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64949414 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6408697195520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64949412 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048^‘ì'ÒæO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚451448089752064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445039392556544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445039410559488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6409353558016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚492689181892608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚439759096574260 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6385756276020 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445039410559488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36606722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚451448764117504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚433373340298240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6723184174592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6385101838264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255786852 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144472186 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6408697195520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101556136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485965825705472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6409353558016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚7649 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101555691 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884341028129792e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111314664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64949414 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6408697195520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64949412 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144452148 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36606279 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111314666 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111314664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚5787.517904569 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚492688985939968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451448089752064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0óÏ6Aû2Ó- _wandbruntime‚5788  _runtime‚5788ÇÝMÎñ&ÔëM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255819348 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6410172999168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492751499410432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111328682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6724208420864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64957582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚7649 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492751684864000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433428244428800 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111328682 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6409516623360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445095763598336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445095763598336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433428227389440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255819348 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚451505936597504 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451505262218752 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 % _timestamp‚1.7884341034879477e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101568387 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255799282 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872¶Ùñš1ÕðM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255819348 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6410172999168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492751499410432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111328682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6724208420864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64957582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚7649 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492751684864000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433428244428800 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111328682 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6409516623360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445095763598336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445095763598336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433428227389440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255819348 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚451505936597504 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451505262218752 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 % _timestamp‚1.7884341034879477e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101568387 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255799282 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944ôS¯ê$2Ö- _wandbruntime‚5788  _runtime‚5788}÷YY×Ó /loss/FRt2/detection_loss‚2.4460723879544632 1loss/SRqqt1/detection_loss‚0.9999807010845918 /loss/FBt3/detection_loss‚1.9519197523593903 2loss/SRqqt4/assignment_loss‚1.4287618590449844 trainer/global_step‚7649 0loss/FBt1/assignment_loss‚0.1928313670856575 /loss/FRt1/detection_loss‚2.0576734976923983 0loss/SRqqt4/detection_loss‚1.811964167488946 0loss/FRt4/assignment_loss‚3.0359814289288645 /loss/FRt3/detection_loss‚2.9551011381610746 1loss/SRqqt3/detection_loss‚2.3759215044624664 epoch‚6 % _timestamp‚1.7884341034895318e+09 _step‚629 /loss/FRt3/assignment_loss‚3.129212266014468 1loss/FBt2/assignment_loss‚0.01682831069774693 /loss/FBt2/detection_loss‚3.5736987367272377 .loss/FBt4/detection_loss‚3.230999997683934 _runtime‚5788.194168873 0loss/FRt1/assignment_loss‚1.3508415818214417 1loss/SRqqt1/assignment_loss‚1.014064496039952 2loss/SRqqt3/assignment_loss‚1.7530511790776955 /loss/FBt3/assignment_loss‚0.291469018207863 .loss/FRt4/detection_loss‚2.248729190765283 /loss/FBt1/detection_loss‚0.7563744950882016 /loss/FRt2/assignment_loss‚2.819443269916203 0loss/SRqqt2/detection_loss‚2.065590683288044 %loss/total_loss‚43.09523457429491 2loss/SRqqt2/assignment_loss‚1.3828940747512712 1loss/FBt4/assignment_loss‚0.20582946995273232’e§Ã^ØØ 1loss/SRqqt1/assignment_loss‚1.014064496039952 2loss/SRqqt3/assignment_loss‚1.7530511790776955 /loss/FBt3/assignment_loss‚0.291469018207863 .loss/FRt4/detection_loss‚2.248729190765283 /loss/FBt1/detection_loss‚0.7563744950882016 /loss/FRt2/assignment_loss‚2.819443269916203 0loss/SRqqt2/detection_loss‚2.065590683288044 %loss/total_loss‚43.09523457429491 2loss/SRqqt2/assignment_loss‚1.3828940747512712 1loss/FBt4/assignment_loss‚0.20582946995273232 /loss/FRt2/detection_loss‚2.4460723879544632 1loss/SRqqt1/detection_loss‚0.9999807010845918 /loss/FBt3/detection_loss‚1.9519197523593903 2loss/SRqqt4/assignment_loss‚1.4287618590449844 trainer/global_step‚7649 0loss/FBt1/assignment_loss‚0.1928313670856575 /loss/FRt1/detection_loss‚2.0576734976923983 0loss/SRqqt4/detection_loss‚1.811964167488946 0loss/FRt4/assignment_loss‚3.0359814289288645 /loss/FRt3/detection_loss‚2.9551011381610746 1loss/SRqqt3/detection_loss‚2.3759215044624664 epoch‚6 % _timestamp‚1.7884341034895318e+09 _step‚629 /loss/FRt3/assignment_loss‚3.129212266014468 1loss/FBt2/assignment_loss‚0.01682831069774693 /loss/FBt2/detection_loss‚3.5736987367272377 .loss/FBt4/detection_loss‚3.230999997683934 _runtime‚5788.194168873 0loss/FRt1/assignment_loss‚1.3508415818214417õ¢€L“Ù‚(:ˆ ¹­åÔýþ’Ž gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.5238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ÛL0Ú‚(:¥ ¹­åÔÄ¢ˆ‘# disk.dm-0.out‚34434.04052734375 disk.sda.in‚0 network.recv‚ 2.013366e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 227.68359375 disk.rbd1.out‚ 3.06640625" disk.nvme0c0n1.in‚ 601.26953125( disk.nvme0c0n1.out‚34434.04052734375 disk.nvme0n1.in‚ 601.26953125 disk.sda2.out‚0$ memory_percent‚94.20351982116699) proc.memory.percent‚17.53401756286621 cpu‚25.354091886563943 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 network.sent‚ 1.7788205e+07 disk.rbd1.in‚0 disk.sda1.in‚0) disk./.usagePercent‚73.87588192547486$ disk./.usageGB‚1100.804027557373& disk.nvme0n1.out‚34434.04052734375 disk.rbd0.out‚0 disk.sda3.out‚ 227.68359375 disk.drbd1777.out‚0 disk.dm-0.in‚ 601.26953125 disk.sda.out‚ 227.68359375 disk.rbd3.out‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 949.6953125 proc.cpu.threads‚17 disk.sda2.in‚0! disk.rbd2.out‚163.12158203125 disk.sda3.in‚0" proc.memory.rssMB‚ 2872.7734375ý¯“Û‚(:ˆ È­åÔî´’Ž gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.133' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö߆„.Ü‚(:£ È­åÔ¼ÌÙ& proc.memory.availableMB‚ 949.640625) proc.memory.percent‚17.53401756286621 disk.sda1.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 601.26953125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 cpu‚25.35331294218638 disk.dm-1.out‚ 228.44921875( disk.nvme0c0n1.out‚34533.35205078125 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17" proc.memory.rssMB‚ 2872.7734375$ disk./.usageGB‚1100.804054260254 disk.rbd0.in‚0! disk.rbd2.out‚164.71142578125 disk.sda3.out‚ 228.44921875) disk./.usagePercent‚73.87588371752759# disk.dm-0.out‚34533.35205078125 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚94.20385360717773 disk.dm-0.in‚ 601.26953125 disk.sda.out‚ 228.44921875& disk.nvme0n1.out‚34533.35205078125" disk.nvme0c0n1.in‚ 601.26953125 disk.rbd2.in‚0 network.sent‚ 1.7896277e+07 network.recv‚ 2.018707e+06 disk.rbd1.out‚ 3.06640625[F9œÝ‚(:‘ ×­åÔ©íŽ gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.54' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.907999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ†YT$Þ‚(:™ ×­åÔ³µ± disk.rbd3.out‚0 disk.sda.in‚0& disk.nvme0n1.out‚34536.59423828125 disk.rbd0.in‚0! disk.rbd2.out‚164.71142578125 disk.sda1.out‚0 disk.dm-0.in‚ 601.26953125 disk.sda1.in‚0" memory_percent‚94.203782081604( proc.memory.availableMB‚ 949.65234375 disk.rbd0.out‚0# disk.dm-0.out‚34536.59423828125 disk.dm-1.in‚0 disk.dm-1.out‚229.3125 disk.rbd1.in‚0" proc.memory.rssMB‚ 2872.7734375) proc.memory.percent‚17.53401756286621 network.sent‚ 1.7901613e+07 disk.sda3.in‚0 disk.sda3.out‚229.3125 disk.drbd1777.out‚0 disk.sda.out‚229.3125 disk.nvme0n1.in‚ 601.26953125" disk.nvme0c0n1.in‚ 601.26953125 disk.sda2.in‚0 network.recv‚ 2.022819e+06 disk.rbd1.out‚ 3.06640625 cpu‚25.352494768878536% disk./.usageGB‚1100.8069038391113 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚34536.59423828125) disk./.usagePercent‚73.87607495515567 disk.rbd3.in‚0‚ŠQ2ß- _wandbruntime‚5822  _runtime‚5822Üw@¯ÇàÁ _runtime‚5822.520748888 _step‚630 & lr-AdamW/pg1‚7.949941836240672e-05 & lr-AdamW/pg2‚7.949941836240672e-05 trainer/global_step‚7699 % _timestamp‚1.7884341378158672e+09N—ÌáÆ & lr-AdamW/pg1‚7.949941836240672e-05 & lr-AdamW/pg2‚7.949941836240672e-05 trainer/global_step‚7699 % _timestamp‚1.7884341378158672e+09 _runtime‚5822.520748888 _step‚630öÆw]2â- _wandbruntime‚5822  _runtime‚5822 NzÓç'ãáO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454306727500288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6450339363840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257413704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚464 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6449683001344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884341378166695e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚454307401865728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454306727500288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7699 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚489048338648576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6766044175872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112015464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚489048362588672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6426591615428 _runtime‚5822.521542744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65357814 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145378200 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65357812 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6425937177672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447857044498944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495814382824448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145398238 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112015466 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6449683001344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102191114 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36833302 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚631 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447857062501888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65357814 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145378200 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65357812 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6425937177672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447857044498944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495814382824448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145398238 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112015466 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6449683001344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102191114 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36833302 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚631 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447857062501888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454306727500288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6450339363840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257413704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚464 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6449683001344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756÷íG}U2å- _wandbruntime‚5823  _runtime‚5823ÚUÑ”æêM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚442600859007400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884341384842207e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6450502429184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6767068141568 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65365980 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112029480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36837844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚442600187530076 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145416718 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257446200 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ZFDeviceStatsMonitor.on_train_bat@\ch_end/allocated_bytes.large_pool.freed‚447913397537792 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6450502429184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495877081467904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6426753612636 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚ 5823.18948945 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447913415540736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65365982 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102204281 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489109989386240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454364574345728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145396654 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447913415540736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112029482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚495876887625728 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚489110013326336 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚436173450956800 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6451158804992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102203824µHí/õ&çïM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚442600859007400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884341384842207e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6450502429184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6767068141568 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65365980 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112029480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36837844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚442600187530076 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145416718 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257446200 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚447913397537792 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6450502429184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495877081467904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6426753612636 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚ 5823.18948945 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447913415540736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65365982 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102204281 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489109989386240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454364574345728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145396654 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447913415540736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112029482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚495876887625728 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚489110013326336 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚436173450956800 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6451158804992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102203824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096øo³1+2è- _wandbruntime‚5823  _runtime‚5823ò-ƒ^éØ 0loss/FRt1/assignment_loss‚1.3489631836613019 /loss/FRt3/detection_loss‚3.3275200596877506 1loss/SRqqt3/detection_loss‚2.2554668203378334 epoch‚6 0loss/FRt2/assignment_loss‚2.8750902566255307 0loss/FRt3/assignment_loss‚2.8437575726282027 /loss/FBt2/detection_loss‚2.1030656354767934 trainer/global_step‚7699 1loss/FBt4/assignment_loss‚0.04694273229688406 /loss/FRt1/detection_loss‚1.5471563199535012 /loss/FRt4/detection_loss‚2.1723555527454197 2loss/SRqqt4/assignment_loss‚1.4057293343875144 1loss/FBt1/assignment_loss‚0.13220643161864293 2loss/FBt3/assignment_loss‚0.010156124830245972 1loss/SRqqt4/detection_loss‚1.6761119740980643 .loss/FBt4/detection_loss‚4.532951354980469 _runtime‚5823.190514762 2loss/SRqqt2/assignment_loss‚1.4768670371310277 1loss/FBt2/assignment_loss‚0.17851496216878462 _step‚633 2loss/SRqqt1/assignment_loss‚0.9374026719170312 1loss/SRqqt3/assignment_loss‚1.436237572095333 0loss/SRqqt2/detection_loss‚1.885368214412169 .loss/FBt1/detection_loss‚0.574712961444415 %loss/total_loss‚43.20661578344827 /loss/FRt4/assignment_loss‚2.731725663639778 /loss/FRt2/detection_loss‚2.3632492516555037 1loss/SRqqt1/detection_loss‚0.9435440222422282 .loss/FBt3/detection_loss‚4.401520073413849 % _timestamp‚1.7884341384859116e+09Ü 7cêÝ 2loss/SRqqt2/assignment_loss‚1.4768670371310277 1loss/FBt2/assignment_loss‚0.17851496216878462 _step‚633 2loss/SRqqt1/assignment_loss‚0.9374026719170312 1loss/SRqqt3/assignment_loss‚1.436237572095333 0loss/SRqqt2/detection_loss‚1.885368214412169 .loss/FBt1/detection_loss‚0.574712961444415 %loss/total_loss‚43.20661578344827 /loss/FRt4/assignment_loss‚2.731725663639778 /loss/FRt2/detection_loss‚2.3632492516555037 1loss/SRqqt1/detection_loss‚0.9435440222422282 .loss/FBt3/detection_loss‚4.401520073413849 % _timestamp‚1.7884341384859116e+09 0loss/FRt1/assignment_loss‚1.3489631836613019 /loss/FRt3/detection_loss‚3.3275200596877506 1loss/SRqqt3/detection_loss‚2.2554668203378334 epoch‚6 0loss/FRt2/assignment_loss‚2.8750902566255307 0loss/FRt3/assignment_loss‚2.8437575726282027 /loss/FBt2/detection_loss‚2.1030656354767934 trainer/global_step‚7699 1loss/FBt4/assignment_loss‚0.04694273229688406 /loss/FRt1/detection_loss‚1.5471563199535012 /loss/FRt4/detection_loss‚2.1723555527454197 2loss/SRqqt4/assignment_loss‚1.4057293343875144 1loss/FBt1/assignment_loss‚0.13220643161864293 2loss/FBt3/assignment_loss‚0.010156124830245972 1loss/SRqqt4/detection_loss‚1.6761119740980643 .loss/FBt4/detection_loss‚4.532951354980469 _runtime‚5823.190514762ù.Œˆpë‚(:’ æ­åÔõíÖ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.89' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.177999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0klÕ((ì‚(: æ­åÔÐÜ• disk.sda2.out‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 cpu‚25.351722573363347 disk.dm-0.in‚ 601.26953125 network.sent‚ 1.7904976e+07% disk./.usageGB‚1100.8069534301758! disk.rbd2.out‚164.78955078125 disk.sda3.out‚ 229.58984375 disk.drbd1777.in‚0 disk.dm-1.out‚ 229.58984375% disk.nvme0c0n1.out‚34636.24609375) disk./.usagePercent‚73.87607828325362 disk.dm-0.out‚34636.24609375 disk.rbd3.in‚0 disk.rbd1.in‚0 network.recv‚ 2.025006e+06 disk.rbd2.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 3.06640625$ memory_percent‚94.20266151428223' proc.memory.availableMB‚ 949.8359375" proc.memory.rssMB‚ 2872.7734375 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd0.in‚0) proc.memory.percent‚17.53401756286621 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda.out‚ 229.58984375 disk.nvme0n1.in‚ 601.26953125# disk.nvme0n1.out‚34636.24609375" disk.nvme0c0n1.in‚ 601.26953125Æ‘Xø“í‚(:ˆ õ­åÔÓ“Ý gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚161.776' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘brÎ&î‚(:› õ­åÔªú disk.dm-0.out‚34735.30078125 disk.sda1.out‚0 disk.rbd2.in‚0! disk.rbd2.out‚164.78955078125 disk.sda3.out‚ 230.2734375 disk.sda1.in‚0 disk.rbd1.out‚ 3.06640625 cpu‚25.350910327160875 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 230.2734375 disk.dm-1.in‚0 disk.nvme0n1.in‚ 601.26953125" disk.nvme0c0n1.in‚ 601.26953125# proc.memory.rssMB‚ 2795.42578125 network.sent‚ 1.8011963e+07) disk./.usagePercent‚73.87607981929882 disk.sda3.in‚0 disk.dm-0.in‚ 601.26953125 network.recv‚ 2.029762e+06 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd1.in‚0% disk.nvme0c0n1.out‚34735.30078125 disk.sda2.out‚0$ memory_percent‚93.72844696044922% disk./.usageGB‚1100.8069763183594 disk.dm-1.out‚ 230.2734375& proc.memory.availableMB‚ 1027.53125 disk.drbd1777.out‚0 disk.rbd3.in‚0# disk.nvme0n1.out‚34735.30078125 disk.sda2.in‚0* proc.memory.percent‚17.061924934387207 proc.cpu.threads‚17 disk.rbd0.out‚0KM}^2ï- _wandbruntime‚5857  _runtime‚5857,süÇðÁ % _timestamp‚1.7884341727222373e+09 _runtime‚5857.426981608 _step‚634 & lr-AdamW/pg1‚7.794174056013498e-05 & lr-AdamW/pg2‚7.794174056013498e-05 trainer/global_step‚7749y)MÌñÆ & lr-AdamW/pg2‚7.794174056013498e-05 trainer/global_step‚7749 % _timestamp‚1.7884341727222373e+09 _runtime‚5857.426981608 _step‚634 & lr-AdamW/pg1‚7.794174056013498e-05ú´eŽ¸2ò- _wandbruntime‚5857  _runtime‚5857öͼ`å óáO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6491325169664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚5857.427849203 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492130899471872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚7749 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457166039613952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457165365248512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚450674714444288 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6809239336960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445330508831960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112716266 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445331180309076 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6491325169664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37060620 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65766214 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457165365248512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457166039613952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438863736314880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6490668807168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6490668807168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146304252 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492130875531776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102826834 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚498940138808832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6466772517080 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146324290 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚450674696441344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚450674714444288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438863753354240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884341727229507e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102826402 QIDeviceStatsMonitor.on_train_batch_start/inacti„úve_split.small_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37060190 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6467426954836 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6809077810176 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112716266 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445331180309076 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6491325169664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37060620 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65766214 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457165365248512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457166039613952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438863736314880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6490668807168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6490668807168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146304252 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492130875531776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102826834 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚498940138808832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6466772517080 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146324290 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚450674696441344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚450674714444288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438863753354240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884341727229507e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102826402 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37060190 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6467426954836 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6809077810176 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445386229397484 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146342770 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65774382 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚499002450361856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492192526269440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 259073052 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚457223212093952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102839015 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚636 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6492144610816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚450731049480192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6491488235008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6467588952044 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112730280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259052986 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112730280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚457222537715200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450731067483136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65774380 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259052986 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499002650495488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚5858.080536717 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445386900874808 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146342770 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6810100285952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259073052 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37065090 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112730282 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884341733754299e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146322706 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438918657484800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492192550209536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102839472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6491488235008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438918640445440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445386229397484 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146342770 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65774382 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚499002450361856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492192526269440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 259073052 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚457223212093952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102839015 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚636 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6492144610816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚450731049480192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6491488235008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6467588952044 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112730280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259052986 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112730280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚457222537715200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450731067483136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65774380 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259052986 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499002650495488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚5858.080536717 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445386900874808 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146342770 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6810100285952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259073052 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37065090 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112730282 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884341733754299e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146322706 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438918657484800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492192550209536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102839472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6491488235008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438918640445440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113417066 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113417064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453492366386688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6852176283648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788434207901022e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6532310975488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147230304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147230304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448116550699368 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460024002996736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147250342 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453492348383744 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147250342 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495213412414976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66174612 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113417066 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37285783 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6531654612992g§ÚÀê'„äO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66174614 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6507607856488 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚639 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453492348383744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260667408 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103460824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6852023145472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113417064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6532310975488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103460395 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚441608959882240 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460024677362176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460024677362176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113417066 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113417064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453492366386688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6852176283648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788434207901022e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6532310975488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147230304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147230304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448116550699368 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460024002996736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147250342 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453492348383744 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147250342 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495213412414976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66174612 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113417066 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37285783 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6531654612992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460024002996736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6508262294244 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6509078729416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502127944440320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7799 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460081849842176 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103473104 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6533130416640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460081175463424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6532474040832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147248758 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441663846973440 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113431080 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103473557 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113431082 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884342085555363e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260699904 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441663864012800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260699904 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260679838 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113431082 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502128131991040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147268822 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66182782 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6532474040832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453548719425536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚453548701422592 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6508424291452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚5893.260950663 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147248758 P¡Úö&‡ðM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884342085555363e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260699904 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441663864012800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260699904 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260679838 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113431082 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502128131991040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147268822 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66182782 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6532474040832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453548719425536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚453548701422592 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6508424291452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚5893.260950663 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147248758 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6852881287680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6509078729416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502127944440320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7799 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460081849842176 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103473104 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6533130416640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460081175463424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6532474040832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147248758 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441663846973440 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113431080 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103473557 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113431082 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37512229 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462882640744960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚450902592566776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104095241 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6895100496384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚498295949298176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚7849 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148176394 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚643 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6572640418816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262274220 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6548443195896 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6894945261056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498295973238272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6573296781312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114117864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114117866 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456310000326144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148156356 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚5927.754392547 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 % _timestamp‚1.7884342430494583e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462882640744960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148156356 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444354166410240n¹Ô–æO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6572640418816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37512229 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462882640744960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7@¦R–m"DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚450902592566776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104095241 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6895100496384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚498295949298176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚7849 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148176394 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚643 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6572640418816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262274220 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6548443195896 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6894945261056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498295973238272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6573296781312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114117864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114117866 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456310000326144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148156356 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚5927.754392547 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 % _timestamp‚1.7884342430494583e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462882640744960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148156356 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444354166410240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6573296781312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6549097633652 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚456310018329088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148176394 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114117866 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚462883315110400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37512665 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚450903264043892 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚444409070540800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456366371367936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚644 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114131882 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148194874 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66591180 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498357623975936 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148194874 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚505253396555776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148174810 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6549259630860 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884342437042842e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6573459846656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚462939813211648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6574116222464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6574116222464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262306690 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104107891 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚7849 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456366353364992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114131882 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6549914068824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262326756 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37516711 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444409053501440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚462940487590400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚450958984609624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6574116222464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6574116222464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262306690 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104107891 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚7849 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456366353364992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114131882 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6549914068824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262326756 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37516711 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444409053501440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚462940487590400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚450958984609624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚444409070540800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456366371367936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚644 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0„"RÖ2š- _wandbruntime‚5928  _runtime‚5928½€ß&Z›Ô /loss/SRqqt4/detection_loss‚1.93123829431004 /loss/FBt1/detection_loss‚0.5932310676822943 0loss/FRt1/assignment_loss‚1.1126490148405235 trainer/global_step‚7849 0loss/FRt4/assignment_loss‚3.2940837775959686 .loss/FRt1/detection_loss‚1.775427699399491 /loss/FRt2/assignment_loss‚2.229276364704348 0loss/FRt3/assignment_loss‚2.9225369691848755 2loss/SRqqt1/assignment_loss‚1.0922542501460104 2loss/SRqqt3/assignment_loss‚1.7891101302845138 /loss/FRt3/detection_loss‚2.7215950406830887 .loss/FBt3/detection_loss‚2.769094154238701 %loss/total_loss‚41.04307186060605 2loss/SRqqt2/assignment_loss‚1.2092775330906187 0loss/SRqqt3/detection_loss‚2.344216171332768 /loss/FBt4/detection_loss‚2.8701364398002625 $ _timestamp‚1.788434243705916e+09 1loss/FBt2/assignment_loss‚0.18686711898772046 1loss/FBt3/assignment_loss‚0.23998549266252667 epoch‚6 _runtime‚5928.410544968 _step‚645 2loss/SRqqt4/assignment_loss‚1.5021775244838662 0loss/FBt4/assignment_loss‚0.3519118227995932 /loss/FRt2/detection_loss‚2.1219324137804643 /loss/FRt4/detection_loss‚2.2868217668112587 1loss/SRqqt1/detection_loss‚1.1482651097809566 .loss/FBt2/detection_loss‚2.750954306125641 1loss/SRqqt2/detection_loss‚1.6833809831124895 1loss/FBt1/assignment_loss‚0.11664841476802898˜Å_œÙ 0loss/FRt4/assignment_loss‚3.2940837775959686 .loss/FRt1/detection_loss‚1.775427699399491 /loss/FRt2/assignment_loss‚2.229276364704348 0loss/FRt3/assignment_loss‚2.9225369691848755 2loss/SRqqt1/assignment_loss‚1.0922542501460104 2loss/SRqqt3/assignment_loss‚1.7891101302845138 /loss/FRt3/detection_loss‚2.7215950406830887 .loss/FBt3/detection_loss‚2.769094154238701 %loss/total_loss‚41.04307186060605 2loss/SRqqt2/assignment_loss‚1.2092775330906187 0loss/SRqqt3/detection_loss‚2.344216171332768 /loss/FBt4/detection_loss‚2.8701364398002625 $ _timestamp‚1.788434243705916e+09 1loss/FBt2/assignment_loss‚0.18686711898772046 1loss/FBt3/assignment_loss‚0.23998549266252667 epoch‚6 _runtime‚5928.410544968 _step‚645 2loss/SRqqt4/assignment_loss‚1.5021775244838662 0loss/FBt4/assignment_loss‚0.3519118227995932 /loss/FRt2/detection_loss‚2.1219324137804643 /loss/FRt4/detection_loss‚2.2868217668112587 1loss/SRqqt1/detection_loss‚1.1482651097809566 .loss/FBt2/detection_loss‚2.750954306125641 1loss/SRqqt2/detection_loss‚1.6833809831124895 1loss/FBt1/assignment_loss‚0.11664841476802898 /loss/SRqqt4/detection_loss‚1.93123829431004 /loss/FBt1/detection_loss‚0.5932310676822943 0loss/FRt1/assignment_loss‚1.1126490148405235 trainer/global_step‚7849…’Ñû“‚(:ˆ Ï®åÔÈ€Ž gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«i½ž‚(:¨ Ï®åÔÈÒÊ) disk./.usagePercent‚73.87605780265088 disk.rbd3.out‚0 disk.nvme0n1.in‚ 601.26953125" disk.nvme0c0n1.in‚ 601.26953125$ memory_percent‚93.73347759246826 cpu‚25.346324840570702 disk.rbd2.in‚0 network.sent‚ 1.824895e+07 disk.dm-1.out‚ 233.83984375 disk.rbd0.out‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 1026.70703125 network.recv‚ 2.055791e+06# disk.dm-0.out‚35232.40185546875 disk.sda1.out‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚35232.40185546875 disk.sda3.out‚ 233.83984375! disk.rbd2.out‚166.77099609375# proc.memory.rssMB‚ 2795.46484375 disk.rbd3.in‚0 disk.dm-0.in‚ 601.26953125 d#2VÆvisk.rbd1.out‚ 3.06640625 disk.sda2.in‚0) proc.memory.percent‚17.06216335296631 proc.cpu.threads‚17 disk.sda.out‚ 233.83984375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚35232.40185546875 disk.drbd1777.in‚0% disk./.usageGB‚1100.8066482543945 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd1777.out‚0¦„OΟ‚(:’ Þ®åÔãºË gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚96.692' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.338399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ«31 ‚(:¦ Þ®åÔ¨õ disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 601.26953125 disk.rbd2.in‚0 disk.sda1.out‚0$ memory_percent‚93.73219013214111) proc.memory.availableMB‚ 1026.91796875) disk./.usagePercent‚73.87618657444061 disk.sda2.out‚0! disk.rbd2.out‚166.84912109375 disk.sda3.in‚0 network.sent‚ 1.8355844e+07( disk.nvme0c0n1.out‚35332.51318359375 disk.sda2.in‚0# disk.dm-0.out‚35332.51318359375 proc.memory.rssMB‚ 2795.46875) proc.memory.percent‚17.06218719482422% disk./.usageGB‚1100.8085670471191 disk.nvme0n1.in‚ 601.26953125 disk.dm-0.in‚ 601.26953125 disk.dm-1.out‚ 234.39453125 cpu‚25.345578861655063 disk.rbd0.out‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd3.in‚0 network.recv‚ 2.060761e+06 disk.rbd1.out‚ 3.06640625 disk.sda3.out‚ 234.39453125 disk.rbd3.out‚0 disk.sda.out‚ 234.39453125 disk.sda1.in‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚35332.51318359375 disk.sda.in‚0¢LE2¡- _wandbruntime‚5962  _runtime‚5962ðÏäÇ¢Á & lr-AdamW/pg1‚7.331422378463847e-05 & lr-AdamW/pg2‚7.331422378463847e-05 trainer/global_step‚7899 % _timestamp‚1.7884342780726883e+09 _runtime‚5962.777447643 _step‚646@!ÊÌ£Æ _step‚646 & lr-AdamW/pg1‚7.331422378463847e-05 & lr-AdamW/pg2‚7.331422378463847e-05 trainer/global_step‚7899 % _timestamp‚1.7884342780726883e+09 _runtime‚5962.777447643†;çš2¤- _wandbruntime‚5962  _runtime‚5962°Üüç'¥áO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114818664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182498304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465741952858624 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465741278493184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37738433 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚447099372938240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚7899 % _timestamp‚1.7884342780734355e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149102446 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104729847 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114818664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚508316352050176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149082408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚5962.778321419 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66991412 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149102446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465741952858624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6614282587136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465741278493184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37738433 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚447099372938240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚7899 % _timestamp‚1.7884342780734355e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149102446 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104729847 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114818664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚508316352050176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149082408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚5962.778321419 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66991412 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149102446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465741952858624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6614282587136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182498304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465741952858624‡)GÂ2§- _wandbruntime‚5963  _runtime‚5963BQ°”s&¨ëM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚501440160859136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37742459 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚7899 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149100862 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263933542 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6590749408232 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465799125338624 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114832682 % _timestamp‚1.7884342787395427e+09 _step‚648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459184005307392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6590094970268 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104742039 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6938709714944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149100862 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453745026477032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6614445652480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459184023310336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453744354999708 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚5963.444798203 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465798450959872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263953608 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104742502 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508379046767616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114832680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6615102028288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447154277068800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚501440136919040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149120926 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6938885908480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114832682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615102028288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263933542 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚459184023310336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66999580 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149120926 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263953608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6614445652480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚447154260029440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465799125338624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66999582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_½§Ëþ~train_batch_end/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 8682209287² ö&©ðM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚501440160859136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37742459 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚7899 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149100862 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263933542 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6590749408232 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465799125338624 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114832682 % _timestamp‚1.7884342787395427e+09 _step‚648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459184005307392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6590094970268 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104742039 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6938709714944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149100862 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453745026477032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6614445652480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459184023310336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453744354999708 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚5963.444798203 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465798450959872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263953608 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104742502 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508379046767616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114832680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6615102028288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447154277068800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚501440136919040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149120926 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6938885908480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114832682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615102028288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263933542 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚459184023310336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66999580 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149120926 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263953608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6614445652480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚447154260029440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465799125338624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66999582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928ˆ«³R¼2ª- _wandbruntime‚5963  _runtime‚5963r!Z«Ô %loss/total_loss‚40.24897625279448 epoch‚6 $ _timestamp‚1.788434278741225e+09 0loss/FRt4/assignment_loss‚2.9738084437946477 1loss/SRqqt2/detection_loss‚1.7257705022368515 1loss/SRqqt3/detection_loss‚2.6438922697100145 /loss/FRt2/assignment_loss‚2.444774057806992 1loss/FBt4/assignment_loss‚0.29313091716418666 2loss/SRqqt4/assignment_loss‚1.5829712659230126 .loss/FRt3/detection_loss‚3.146871400916058 /loss/FRt4/detection_loss‚1.8827755283565049 0loss/FRt3/assignment_loss‚3.4803839351819907 0loss/FBt1/assignment_loss‚0.0851262259852863 -loss/FBt2/detection_loss‚2.30979945404189 trainer/global_step‚7899 _runtime‚5963.445869241 /loss/FBt3/detection_loss‚2.0437975029150643 2loss/SRqqt2/assignment_loss‚1.5087097535530727 /loss/FRt1/detection_loss‚1.3649366837844514 .loss/FRt2/detection_loss‚2.248121612682575 1loss/SRqqt1/detection_loss‚1.0282489228993654 /loss/FBt1/detection_loss‚0.4490732236480787 _step‚649 2loss/SRqqt1/assignment_loss‚0.8656085054519482 2loss/SRqqt3/assignment_loss‚1.8543441891670227 .loss/FBt4/detection_loss‚2.620732858777046 1loss/FBt2/assignment_loss‚0.11887835633076195 0loss/FBt3/assignment_loss‚0.5035778175806627 1loss/SRqqt4/detection_loss‚1.5646298632902258 0loss/FRt1/assignment_loss‚1.5090129615967733LÈKŒ_¬Ù 1loss/FBt2/assignment_loss‚0.11887835633076195 0loss/FBt3/assignment_loss‚0.5035778175806627 1loss/SRqqt4/detection_loss‚1.5646298632902258 0loss/FRt1/assignment_loss‚1.5090129615967733 %loss/total_loss‚40.24897625279448 epoch‚6 $ _timestamp‚1.788434278741225e+09 0loss/FRt4/assignment_loss‚2.9738084437946477 1loss/SRqqt2/detection_loss‚1.7257705022368515 1loss/SRqqt3/detection_loss‚2.6438922697100145 /loss/FRt2/assignment_loss‚2.444774057806992 1loss/FBt4/assignment_loss‚0.29313091716418666 2loss/SRqqt4/assignment_loss‚1.5829712659230126 .loss/FRt3/detection_loss‚3.146871400916058 /loss/FRt4/detection_loss‚1.8827755283565049 0loss/FRt3/assignment_loss‚3.4803839351819907 0loss/FBt1/assignment_loss‚0.0851262259852863 -loss/FBt2/detection_loss‚2.30979945404189 trainer/global_step‚7899 _runtime‚5963.445869241 /loss/FBt3/detection_loss‚2.0437975029150643 2loss/SRqqt2/assignment_loss‚1.5087097535530727 /loss/FRt1/detection_loss‚1.3649366837844514 .loss/FRt2/detection_loss‚2.248121612682575 1loss/SRqqt1/detection_loss‚1.0282489228993654 /loss/FBt1/detection_loss‚0.4490732236480787 _step‚649 2loss/SRqqt1/assignment_loss‚0.8656085054519482 2loss/SRqqt3/assignment_loss‚1.8543441891670227 .loss/FBt4/detection_loss‚2.620732858777046‰Á縢­‚(:’ í®åÔÏŽé gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.52' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.104000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ÊÑ®‚(:“ í®åÔúø  network.recv‚ 2.065287e+06 disk.sda2.out‚0! disk.dm-0.out‚35332.849609375 network.sent‚ 1.836217e+07 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 3.06640625 proc.memory.rssMB‚ 2795.46875) disk./.usagePercent‚73.87618657444061$ disk.nvme0n1.out‚35332.849609375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.out‚ 235.21875% disk./.usageGB‚1100.8085670471191 disk.sda3.out‚ 235.21875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0) proc.memory.percent‚17.06218719482422& disk.nvme0c0n1.out‚35332.849609375 disk.rbd0.out‚0 disk.rbd2.in‚0! disk.rbd2.out‚168.36865234375 disk.sda.in‚0" disk.nvme0c0n1.in‚ 601.26953125 disk.rbd0.in‚0 disk.dm-0.in‚ 601.26953125 disk.sda.out‚ 235.21875 disk.rbd1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0$ memory_percent‚93.73273849487305' proc.memory.availableMB‚ 1026.828125 cpu‚25.34484508817423 disk.nvme0n1.in‚ 601.26953125ýú|A‘¯‚(:† ü®åÔå¹€Ž gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚90.755' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.151 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãÄ5)°‚(:ž ü®åÔ§¼•‘ network.sent‚ 1.8470657e+07 disk.dm-0.in‚ 601.26953125 disk.sda.in‚0 disk.dm-1.out‚ 236.078125) proc.memory.percent‚17.06218719482422 disk.rbd0.out‚0! disk.rbd2.out‚168.44677734375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.06640625" disk.nvme0c0n1.in‚ 601.26953125 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda1.out‚0( disk./.usagePercent‚73.8761947666817% disk./.usageGB‚1100.8086891174316 disk.sda.out‚ 236.078125 disk.rbd0.in‚0 disk.sda3.out‚ 236.078125) proc.memory.availableMB‚ 1027.01953125 proc.memory.rssMB‚ 2795.46875 disk.sda3.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 601.26953125& disk.nvme0n1.out‚35432.13623046875 disk.sda2.in‚0 cpu‚25.34410141961575 network.recv‚ 2.071146e+06 disk.rbd3.out‚0# disk.dm-0.out‚35432.13623046875( disk.nvme0c0n1.out‚35432.13623046875$ memory_percent‚93.73157024383545 proc.cpu.threads‚17ÉžM2±- _wandbruntime‚5997  _runtime‚5997`!BÛDzÁ _runtime‚5997.987814434 _step‚650 & lr-AdamW/pg1‚7.178796193020104e-05 & lr-AdamW/pg2‚7.178796193020104e-05 trainer/global_step‚7949 % _timestamp‚1.7884343132829707e+09Ÿè[Ì³Æ trainer/global_step‚7949 % _timestamp‚1.7884343132829707e+09 _runtime‚5997.987814434 _step‚650 & lr-AdamW/pg1‚7.178796193020104e-05 & lr-AdamW/pg2‚7.178796193020104e-05ŠP5’2´- _wandbruntime‚5997  _runtime‚5997M]\Vç'µáO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6654612030464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚461945304210944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37963849 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6655268392960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 % _timestamp‚1.7884343132837641e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6980785569792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449844579466240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚468599916241408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456475347778708 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265527924 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6630768312468 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105363661 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504461047004672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468599916241408 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456474676301592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105364103 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150028498 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚461945322213888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461945322213888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115519466 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150008460 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚7949 _runtime‚5997.988672831 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468600590606848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504461023064576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6630113874712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67399812‰½©å¶æO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67399812 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6654612030464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚461945304210944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37963849 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6655268392960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 % _timestamp‚1.7884343132837641e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6980785569792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449844579466240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚468599916241408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456475347778708 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265527924 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6630768312468 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105363661 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504461047004672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468599916241408 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456474676301592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105364103 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150028498 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚461945322213888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461945322213888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115519466 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150008460 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚7949 _runtime‚5997.988672831 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468600590606848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504461023064576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6630113874712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605‹EJŒ42·- _wandbruntime‚5998  _runtime‚5998>iï†ñ&¸ëM L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105376356 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462001675252736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6981659965952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105376793 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511504515027456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456530396867116 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚504522673802240 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150046978 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37968376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6981817285120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6655431458304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6630930309676 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265580460 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115533480 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150026914 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150026914 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462001675252736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462001657249792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚449899466557440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468657763086848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265580460 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6655431458304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6631584747640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37968811 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456531068344440 trainer/global_step‚7949 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6656087834112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150046978 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449899483596800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6656087834112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265560394 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚5998.660555289 _step‚652 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115533482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ÷ºö&¹ðM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚5998.660555289 _step‚652 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115533482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105376356 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462001675252736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6981659965952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105376793 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511504515027456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456530396867116 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚504522673802240 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150046978 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37968376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6981817285120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6655431458304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6630930309676 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265580460 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115533480 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150026914 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150026914 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462001675252736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462001657249792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚449899466557440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468657763086848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265580460 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6655431458304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6631584747640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37968811 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456531068344440 trainer/global_step‚7949 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6656087834112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150046978 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449899483596800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6656087834112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265560394 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680ŒǾÀJ2º- _wandbruntime‚5998  _runtime‚5998¸†^»Ø epoch‚6 1loss/FBt1/assignment_loss‚0.07624853875308625 /loss/FRt4/detection_loss‚2.3260400619771744 2loss/SRqqt1/assignment_loss‚0.9679624597233734 2loss/SRqqt3/assignment_loss‚2.1276164723729547 2loss/SRqqt4/assignment_loss‚1.6379110183001362 /loss/FRt3/detection_loss‚3.1595471786416094 &loss/total_loss‚41.197841947657714 trainer/global_step‚7949 /loss/FRt1/detection_loss‚1.4256689299718808 /loss/FRt1/assignment_loss‚1.681011069629152 /loss/FRt2/assignment_loss‚2.751831058944975 /loss/FRt3/assignment_loss‚3.457229531329611 1loss/FBt4/assignment_loss‚0.06651972678810125 1loss/SRqqt1/detection_loss‚0.7709966934692691 1loss/SRqqt3/detection_loss‚2.0332416892051697 .loss/FBt3/detection_loss‚2.016818583011627 /loss/FRt2/detection_loss‚2.4539544745570137 2loss/SRqqt2/assignment_loss‚1.4502604909393253 4loss/FBt2/assignment_loss‚0.00022786019108025357 0loss/FBt3/assignment_loss‚0.2311686559613528 /loss/FBt1/detection_loss‚0.6288142738246982 .loss/FBt4/detection_loss‚2.143240049481392 _step‚653 1loss/SRqqt2/detection_loss‚1.9442730737583978 1loss/SRqqt4/detection_loss‚1.8058084159025125 $ _timestamp‚1.788434313957312e+09 _runtime‚5998.661955938 /loss/FRt4/assignment_loss‚3.087035592397054 .loss/FBt2/detection_loss‚2.954416048526764$ͺc¼Ý 1loss/SRqqt2/detection_loss‚1.9442730737583978 1loss/SRqqt4/detection_loss‚1.8058084159025125 $ _timestamp‚1.788434313957312e+09 _runtime‚5998.661955938 /loss/FRt4/assignment_loss‚3.087035592397054 .loss/FBt2/detection_loss‚2.954416048526764 epoch‚6 1loss/FBt1/assignment_loss‚0.07624853875308625 /loss/FRt4/detection_loss‚2.3260400619771744 2loss/SRqqt1/assignment_loss‚0.9679624597233734 2loss/SRqqt3/assignment_loss‚2.1276164723729547 2loss/SRqqt4/assignment_loss‚1.6379110183001362 /loss/FRt3/detection_loss‚3.1595471786416094 &loss/total_loss‚41.197841947657714 trainer/global_step‚7949 /loss/FRt1/detection_loss‚1.4256689299718808 /loss/FRt1/assignment_loss‚1.681011069629152 /loss/FRt2/assignment_loss‚2.751831058944975 /loss/FRt3/assignment_loss‚3.457229531329611 1loss/FBt4/assignment_loss‚0.06651972678810125 1loss/SRqqt1/detection_loss‚0.7709966934692691 1loss/SRqqt3/detection_loss‚2.0332416892051697 .loss/FBt3/detection_loss‚2.016818583011627 /loss/FRt2/detection_loss‚2.4539544745570137 2loss/SRqqt2/assignment_loss‚1.4502604909393253 4loss/FBt2/assignment_loss‚0.00022786019108025357 0loss/FBt3/assignment_loss‚0.2311686559613528 /loss/FBt1/detection_loss‚0.6288142738246982 .loss/FBt4/detection_loss‚2.143240049481392 _step‚653Ö7wž½‚(:“ ‹¯åÔšéÚ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.156' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.631200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÑ)¾‚(:ž ‹¯åÔ·¸( disk.nvme0c0n1.out‚35531.24169921875 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 3.06640625" disk.nvme0c0n1.in‚ 601.26953125 disk.dm-1.out‚ 236.734375 network.recv‚ 2.074246e+06 disk.rbd3.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.drbd1777.out‚0% disk./.usageGB‚1100.8087196350098 disk.dm-0.in‚ 601.26953125# disk.dm-0.out‚35531.24169921875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0 proc.memory.rssMB‚ 2795.46875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 236.734375 disk.dm-1.in‚0) disk./.usagePercent‚73.87619681474197 disk.sda.out‚ 236.734375 disk.nvme0n1.in‚ 601.26953125& disk.nvme0n1.out‚35531.24169921875 network.sent‚ 1.8475819e+07 disk.sda1.out‚0! disk.rbd2.out‚168.44677734375 disk.drbd1777.in‚0' proc.memory.availableMB‚ 1026.515625 cpu‚25.343337078816557 disk.sda.in‚0$ memory_percent‚93.73464584350586) proc.memory.percent‚17.06218719482422l&Ÿž¿‚(:“ š¯åÔǨ˜Ž gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.329' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.665800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ÙÊb.À‚(:£ š¯åÔĺΑ disk.rbd3.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚35630.46142578125 disk.sda1.in‚0 disk.dm-1.out‚ 237.19921875 proc.cpu.threads‚17) proc.memory.availableMB‚ 1026.37890625 disk.dm-0.in‚ 601.26953125 disk.drbd1777.in‚0) proc.memory.percent‚17.06218719482422 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 168.626953125) disk./.usagePercent‚73.87619681474197 disk.sda.out‚ 237.19921875 disk.dm-1.in‚0 disk.rbd1.out‚ 3.06640625 disk.sda2.in‚0% disk./.usageGB‚1100.8087196350098 disk.sda.in‚0$ memory_percent‚93.73548030853271 proc.memory.rssMB‚ 2795.46875 disk.sda1.out‚0& disk.nvme0n1.out‚35630.46142578125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 237.19921875 disk.drbd1777.out‚0 network.recv‚ 2.078252e+06 disk.rbd1.in‚0( disk.nvme0c0n1.out‚35630.46142578125 cpu‚25.34259550844073 disk.sda3.in‚0 network.sent‚ 1.8582162e+07 disk.nvme0n1.in‚ 601.26953125" disk.nvme0c0n1.in‚ 601.26953125fN~Á‚(:’ ©¯åÔÕáë gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.91' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.581999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oÒÖ)‚(:ž ©¯åÔãÊ•‘ disk.sda1.in‚0" disk.nvme0c0n1.in‚ 601.26953125 disk.rbd0.in‚0$ memory_percent‚93.73538494110107 disk.rbd3.in‚0 disk.dm-1.out‚ 238.046875 disk.nvme0n1.in‚ 601.26953125 cpu‚25.341874785493545 disk.sda.in‚0 disk.rbd1.out‚ 3.06640625 proc.memory.rssMB‚ 2795.46875 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.sda.out‚ 238.046875% disk./.usageGB‚1100.8087272644043 disk.drbd1777.in‚0 network.sent‚ 1.8586441e+07 disk.sda2.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚73.87619732675704 disk.sda3.out‚ 238.046875 disk.drbd1777.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚35630.78076171875( disk.nvme0c0n1.out‚35630.78076171875 disk.rbd2.out‚ 168.626953125 disk.dm-0.in‚ 601.26953125# disk.dm-0.out‚35630.78076171875 disk.rbd1.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1026.39453125) proc.memory.percent‚17.06218719482422 network.recv‚ 2.080728e+06 disk.sda3.in‚0£QP2Ã- _wandbruntime‚6032  _runtime‚6032Jh} ÅÄ¿ % lr-AdamW/pg1‚7.02703534987203e-05 % lr-AdamW/pg2‚7.02703534987203e-05 trainer/global_step‚7999 % _timestamp‚1.7884343481799312e+09 _runtime‚6032.884694281 _step‚654Žc|ÊÅÄ % _timestamp‚1.7884343481799312e+09 _runtime‚6032.884694281 _step‚654 % lr-AdamW/pg1‚7.02703534987203e-05 % lr-AdamW/pg2‚7.02703534987203e-05 trainer/global_step‚7999Žš:ú2Æ- _wandbruntime‚6032  _runtime‚6032“.¦q#ÇàO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471458553989632 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464762974156288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105998624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38190410 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7023927045120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_¦–kÃ$batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚471458553989632 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105998167 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507543559947776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6670949214120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452589768954880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚507543583887872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116220266 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚7999'×Äë'ÈåO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150934512 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚464762956153344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116220264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7023761324032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 $ _timestamp‚1.788434348180648e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150954550 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6696254198784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚655 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452589785994240 _runtime‚6032.885516511 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150934512 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 267154776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6696254198784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150954550 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚514567510932992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6671603651876 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚471459228355072 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464762974156288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67808212 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116220264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚471459228355072 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚471458553989632 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105998167 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507543559947776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6670949214120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452589768954880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚507543583887872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116220266 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚7999 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471458553989632 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464762974156288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105998624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38190410 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7023927045120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106011330 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚464819327195136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150952966 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267207312 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116234280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7024616631808 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116234282 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116234282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67816380 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267207312 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106010878 % _timestamp‚1.7884343488389413e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 267187246 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6672420087048 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67816382 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚464819327195136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7024784436736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6697073639936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6697073639936 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150973030 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116234280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚507605210685440 trainer/global_step‚7999 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464819309192192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452‰ˆã# ËðM _step‚656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267187246 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚464819309192192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6672420087048 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67816382 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚464819327195136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7024784436736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6697073639936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6697073639936 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150973030 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116234280 MCDeviceStatsMonitor.on_train_batch_end/allocation.smn‡æ6êall_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚507605210685440 trainer/global_step‚7999 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464819309192192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚514630019062272 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106011330 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚464819327195136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150952966 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267207312 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116234280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7024616631808 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116234282 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116234282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67816380 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267207312 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106010878 % _timestamp‚1.7884343488389413e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 267187246 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510626120771072 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462047431513524 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455334975482880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68216612 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151860564 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6737240004608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38417900 _step‚659 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467580626098688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚474317866103296 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116921066 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474317191737856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106634514 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚462046760036408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268781628 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6711784553528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151860564 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38417452 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106634064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68216614 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517692812341248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7066715510272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884343833102446e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚474317191737856 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151880602 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6712438991284 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6736583642112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467580608095744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510626120771072 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462047431513524 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455334975482880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68216612 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151860564 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6737240004608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38417900 _step‚659 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467580626098688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚474317866103296 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116921066 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474317191737856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106634514 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚462046760036408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268781628 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6711784553528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151860564 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38417452 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106634064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68216614 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517692812341248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7066715510272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884343833102446e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚474317191737856 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151880602 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_startè´Uóƒ/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6712438991284 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6736583642112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467580608095744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 148930560 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6737403069952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚467636961134592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 172870656 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚660 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106646778 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚474375038583296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268814098 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68224782 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510687747568640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788434383966154e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510687771508736 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚474374364204544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38422444 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455389896652800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7067749978112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6737403069952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116935080 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517755348616192 _runtime‚6068.671476759 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚467636979137536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151899082 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚474375038583296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462103152079256 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106647226 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268834164 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151879018 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116935082 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151879018 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268814098 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0iÖ“õ&ÛïM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116935080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268834164 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151879018 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116935082 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151879018 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268814098 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268834164 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6712600988492 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚462102480601932 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38421998 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6738059445760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68224780 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 148930560 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6737403069952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚467636961134592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 172870656 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚660 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106646778 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚474375038583296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268814098 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68224782 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510687747568640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788434383966154e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510687771508736 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚474374364204544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38422444 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455389896652800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7067749978112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6737403069952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116935080 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517755348616192 _runtime‚6068.671476759 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚467636979137536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151899082 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚474375038583296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462103152079256 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106647226 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6777569447936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68625012 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚663 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477175829486080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7109807775232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152786616 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6778225810432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6777569447936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270428520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38645275 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6102.975578832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38644823 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 270408480 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚464833473380932 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚470398260038144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464832801903816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107270289 % _timestamp‚1.7884344182706203e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚470398278041088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477175829486080 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152806654 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚520818629053440óñ Òì'èæO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38644823 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 270408480 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚464833473380932 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚470398260038144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464832801903816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107270289 % _timestamp‚1.7884344182706203e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚470398278041088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477175829486080 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152806654 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚520818629053440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6777569447936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68625012 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚663 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477175829486080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7109807775232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152786616 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6778225810432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6777569447936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270428520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38645275 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6102.975578832—ÃüÀŠ2é- _wandbruntime‚6103  _runtime‚6103Õ6 ðñ&êëM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152805070 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚8099 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117635882 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107282560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477233676331520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7110678791168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464889193946664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38649829 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚458135086141440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117635882 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270440950 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107283011 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152825134 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6778388875776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513770284451840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270440950 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152825134 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884344189293356e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚458135103180800 _step‚664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270461016 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6779045251584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38649380 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477233676331520 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117635880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68633182 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _runtime‚6103.634607046 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470454613076992 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477233001952768 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470454631079936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6779045251584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚520881142405120 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0- ÇäžëðM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152805070 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚8099 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117635882 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107282560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477233676331520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7110678791168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464889193946664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38649829 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚458135086141440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117635882 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270440950 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107283011 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152825134 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6778388875776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513770284451840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270440950 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152825134 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884344189293356e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚458135103180800 _step‚664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270461016 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6779045251584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38649380 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477233676331520 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117635880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68633182 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _runtime‚6103.634607046 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470454613076992 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477233001952768 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470454631079936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6779045251584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚520881142405120˜_ekU2ì- _wandbruntime‚6103  _runtime‚6103ìKV¦UíÏ /loss/FBt2/assignment_loss‚0.234918876235497 0loss/FBt4/assignment_loss‚0.3915430178333606 .loss/FRt2/detection_loss‚2.460701313047182 /loss/FBt1/detection_loss‚0.4215731926080657 %loss/total_loss‚40.86091742015657 % _timestamp‚1.7884344189312227e+09 /loss/FRt4/assignment_loss‚3.023724369380785 .loss/FRt1/detection_loss‚1.781163633547046 1loss/SRqqt1/detection_loss‚0.9205834748637345 1loss/SRqqt2/detection_loss‚1.6233184698682566 0loss/SRqqt4/detection_loss‚1.856540871914043 /loss/FBt3/detection_loss‚3.7164924293756485 epoch‚6 .loss/FRt2/assignment_loss‚2.74245984071777 /loss/FRt3/assignment_loss‚3.337699966771262 2loss/SRqqt3/assignment_loss‚1.4976653100216184 1loss/FBt1/assignment_loss‚0.09523642374343952 1loss/FBt3/assignment_loss‚0.00289588792657014 0loss/FRt1/assignment_loss‚0.8346693813800812 2loss/SRqqt1/assignment_loss‚0.9967562128763853 2loss/SRqqt4/assignment_loss‚1.2169744811605576 .loss/FRt3/detection_loss‚2.963545747101307 1loss/SRqqt3/detection_loss‚2.6401383013560853 /loss/FBt2/detection_loss‚2.2257985919713974 /loss/FRt4/detection_loss‚2.3528086627307148 .loss/FBt4/detection_loss‚2.184029792036329 trainer/global_step‚8099 _runtime‚ 6103.6361802 2loss/SRqqt2/assignment_loss‚1.3396791716894279 _step‚665yÆgXZîÔ /loss/FRt4/assignment_loss‚3.023724369380785 .loss/FRt1/detection_loss‚1.781163633547046 1loss/SRqqt1/detection_loss‚0.9205834748637345 1loss/SRqqt2/detection_loss‚1.6233184698682566 0loss/SRqqt4/detection_loss‚1.856540871914043 /loss/FBt3/detection_loss‚3.7164924293756485 epoch‚6 .loss/FRt2/assignment_loss‚2.74245984071777 /loss/FRt3/assignment_loss‚3.337699966771262 2loss/SRqqt3/assignment_loss‚1.4976653100216184 1loss/FBt1/assignment_loss‚0.09523642374343952 1loss/FBt3/assignment_loss‚0.00289588792657014 0loss/FRt1/assignment_loss‚0.8346693813800812 2loss/SRqqt1/assignment_loss‚0.9967562128763853 2loss/SRqqt4/assignment_loss‚1.2169744811605576 .loss/FRt3/detection_loss‚2.963545747101307 1loss/SRqqt3/detection_loss‚2.6401383013560853 /loss/FBt2/detection_loss‚2.2257985919713974 /loss/FRt4/detection_loss‚2.3528086627307148 .loss/FBt4/detection_loss‚2.184029792036329 trainer/global_step‚8099 _runtime‚ 6103.6361802 2loss/SRqqt2/assignment_loss‚1.3396791716894279 _step‚665 /loss/FBt2/assignment_loss‚0.234918876235497 0loss/FBt4/assignment_loss‚0.3915430178333606 .loss/FRt2/detection_loss‚2.460701313047182 /loss/FBt1/detection_loss‚0.4215731926080657 %loss/total_loss‚40.86091742015657 % _timestamp‚1.7884344189312227e+09™]¤OÞ“ï‚(:ˆ ô¯åÔßíð gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.3884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x?À&ð‚(:› ô¯åÔð¾ disk.sda.in‚0 disk.dm-1.out‚ 242.2109375 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 network.sent‚ 1.8817029e+07 disk.dm-0.in‚ 671.75390625 disk.rbd1.out‚ 3.06640625" disk.nvme0c0n1.in‚ 671.75390625! disk.rbd2.out‚170.45556640625) proc.memory.percent‚17.06218719482422 disk.nvme0n1.in‚ 671.75390625 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 242.2109375 disk.drbd1777.in‚0! disk.dm-0.out‚36032.212890625 disk.rbd3.in‚0 cpu‚25.338228501190578 network.recv‚ 2.102822e+06' disk./.usagePercent‚73.876339154931 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 2795.46875 disk.sda.out‚ 242.2109375& disk.nvme0c0n1.out‚36032.212890625 disk.rbd0.in‚0$ memory_percent‚93.73633861541748) proc.memory.availableMB‚ 1026.23828125% disk./.usageGB‚1100.8108406066895$ disk.nvme0n1.out‚36032.212890625 disk.sda2.in‚0 proc.cpu.threads‚17²Kš“ñ‚(:ˆ ƒ°åÔ“îÿ gpu.0.gpu‚33.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.133' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôxä&ò‚(:› ƒ°åÔó·²! disk.rbd2.out‚170.45556640625 disk.sda3.out‚ 243.08203125 disk.rbd3.out‚0$ memory_percent‚93.73719692230225) proc.memory.percent‚17.06218719482422 cpu‚25.337485186214636 disk.sda.out‚ 243.08203125 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 906.1953125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 243.08203125 disk.nvme0n1.in‚ 906.1953125$ disk.nvme0n1.out‚36131.419921875 disk.rbd1.out‚ 3.06640625& disk.nvme0c0n1.out‚36131.419921875 disk.rbd0.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 906.1953125! disk.dm-0.out‚36131.419921875) proc.memory.availableMB‚ 1026.09765625 proc.memory.rssMB‚ 2795.46875% disk./.usageGB‚1100.8108406066895 disk.drbd1777.in‚0 disk.sda1.in‚0 network.sent‚ 1.8925812e+07 network.recv‚ 2.109163e+06' disk./.usagePercent‚73.876339154931 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.rbd2.in‚0J?œó‚(:‘ ’°åÔ€úæ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.4' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.880000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0: «.ô‚(:£ ’°åÔô´ñ' disk./.usagePercent‚73.876339154931% disk./.usageGB‚1100.8108406066895 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 917.80078125 disk.sda3.out‚ 243.60546875 disk.sda.out‚ 243.60546875& disk.nvme0n1.out‚36230.68994140625 disk.sda2.out‚0$ memory_percent‚93.73478889465332( proc.memory.availableMB‚ 1026.4921875 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 network.sent‚ 1.8931144e+07 disk.dm-1.out‚ 243.60546875 disk.nvme0n1.in‚ 917.80078125 disk.rbd0.out‚0 proc.memory.rssMB‚ 2795.46875 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚36230.68994140625) proc.memory.percent‚17.06218719482422 disk.rbd0.in‚0! disk.rbd2.out‚172.04541015625 disk.dm-0.in‚ 917.80078125# disk.dm-0.out‚36230.68994140625 disk.rbd1.in‚0 disk.rbd1.out‚ 3.06640625 cpu‚25.336781027297864 network.recv‚ 2.112814e+06ÿhø2õ- _wandbruntime‚6138  _runtime‚6138¢²ø¹ÆöÀ & lr-AdamW/pg1‚6.577260200337261e-05 & lr-AdamW/pg2‚6.577260200337261e-05 trainer/global_step‚8149 $ _timestamp‚1.788434453296948e+09 _runtime‚6138.001674521 _step‚666£{¸Ë÷Å & lr-AdamW/pg1‚6.577260200337261e-05 & lr-AdamW/pg2‚6.577260200337261e-05 trainer/global_step‚8149 $ _timestamp‚1.788434453296948e+09 _runtime‚6138.001674521 _step‚666š}œ´h2ø- _wandbruntime‚6138  _runtime‚6138:ß{ïç'ùáO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6794109670100 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚8149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚473215911980544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 272055372 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚480034467234304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523943885282304 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚467619515248340 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118322664 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118322666 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6819211616256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153732706 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480034467234304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚516791170597376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523944074943488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚460825405578240 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 _runtime‚6138.002555541 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153732706 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚473215911980544 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118322664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107906389 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚480035141599744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6819211616256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6818555253760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69033414 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚480035141599744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884344532976725e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7152714684928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118322666 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6819211616256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153732706 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480034467234304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚516791170597376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523944074943488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚460825405578240 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 _runtime‚6138.002555541 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153732706 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚473215911980544 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118322664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107906389 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚480035141599744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6819211616256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6818555253760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69033414 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚480035141599744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884344532976725e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7152714684928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523943885282304 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚467619515248340 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118322664 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69041580 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153751186 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚8149 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚473272265019392 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6820031057408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚524006379052032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚516852845275136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118336682 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 681937468\ÁÅÚÞ1600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118336682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6794926105272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118336680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272087868 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118336680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153731122 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6794271667308 _step‚668 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7153557716992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69041582 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7153731813376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚480092314079744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6819374681600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚467674564336748 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153751186 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107919156 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153731122 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480091639700992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272087868 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚467674564336748 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153751186 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107919156 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153731122 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480091639700992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272087868 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69041580 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153751186 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚8149 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚473272265019392 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6820031057408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚524006379052032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚516852845275136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118336682 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6819374681600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118336682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6794926105272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118336680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272087868 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118336680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153731122 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6794271667308 _step‚668 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7153557716992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69041582 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7153731813376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚480092314079744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6819374681600œ•r†ý2þ- _wandbruntime‚6138  _runtime‚6138º¦“=^ÿØ &loss/total_loss‚43.874757109925596 1loss/FBt1/assignment_loss‚0.09482529064181987 % _timestamp‚1.7884344539497077e+09 /loss/FBt3/detection_loss‚3.0140706996122995 trainer/global_step‚8149 2loss/SRqqt2/assignment_loss‚1.3598633366278732 1loss/FBt3/assignment_loss‚0.14870767315971042 1loss/SRqqt3/detection_loss‚2.5664076552246557 0loss/FRt4/assignment_loss‚2.9156066050716474 2loss/SRqqt1/assignment_loss‚1.0740734076788347 /loss/FRt3/detection_loss‚2.7611905495325724 0loss/SRqqt4/detection_loss‚1.741044718386798 .loss/FBt4/detection_loss‚2.264614123851061 _runtime‚6138.654338592 0loss/FRt2/assignment_loss‚2.9343305463376255 /loss/FBt1/detection_loss‚0.5440434184089088 /loss/FBt2/detection_loss‚3.0034445077180862 _step‚669 .loss/FRt2/detection_loss‚2.372035607047703 /loss/FRt3/assignment_loss‚3.687516830364863 1loss/SRqqt4/assignment_loss‚1.559463171737975 1loss/FBt4/assignment_loss‚0.43740739244276483 1loss/SRqqt1/detection_loss‚1.0629866581049658 2loss/SRqqt3/assignment_loss‚2.2970173405201146 /loss/FRt1/detection_loss‚1.8410465293563902 1loss/SRqqt2/detection_loss‚1.6872535921060121 epoch‚6 0loss/FRt1/assignment_loss‚1.4119595557451248 0loss/FBt2/assignment_loss‚0.9051372890446601 /loss/FRt4/detection_loss‚2.1907106112031376ÆDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚482893104982528 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476033563922944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚8199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6860197422080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519873731420672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470404885638632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6834290571752 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108542605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6859541059584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463570612106240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6860197422080 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚482893779347968 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚6172.922411868 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69441812 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚482893779347968 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119023466 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6834945009508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7195769371648 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884344882174966e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39101259 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154638720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6859541059584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154658758 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69441814 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154638720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚463570595066880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273662184 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119023466 TGDevic‹åýí‘eStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39100793 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚482893104982528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚476033581925888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚476033563922944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470405557115748 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519873707480576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚482893104982528 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476033563922944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚8199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6860197422080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519873731420672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470404885638632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6834290571752 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108542605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6859541059584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463570612106240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6860197422080 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚482893779347968 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚6172.922411868 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69441812 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚482893779347968 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119023466 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6834945009508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7195769371648 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884344882174966e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39101259 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154638720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6859541059584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154658758 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69441814 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154638720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521ŸÄÔ+¼2‹ - _wandbruntime‚6173  _runtime‚6173㦕Uï&Œ éM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273714720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476089934964736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108555338 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154677238 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154677238 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119037480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚476089916961792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6860360487424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚527131984510464 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273694654 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚527132174158336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519935358218240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463625516236800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚482950951827968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119037482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119037480 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6861016863232 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154657174 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚476089934964736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476089916961792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6860360487424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788434488868601e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273714720 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463625499197440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6835761444680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6835107006716 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154657174 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69449980 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39105808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521Q;’³ô& îM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154677238 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119037480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚476089916961792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6860360487424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚527131984510464 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273694654 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚527132174158336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519935358218240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463625516236800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚482950951827968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119037482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119037480 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6861016863232 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154657174 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚476089934964736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476089916961792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6860360487424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788434488868601e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273714720 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463625499197440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6835761444680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6835107006716 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154657174 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69449980 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39105808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273714720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476089934964736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108555338 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154677238 XM€c2Ž - _wandbruntime‚6173  _runtime‚6173hm  Ò 2loss/SRqqt1/assignment_loss‚1.1203063832736522 1loss/FBt4/assignment_loss‚0.14057852145439634 1loss/SRqqt1/detection_loss‚1.1924523823666122 /loss/FBt1/detection_loss‚0.5042429083071428 .loss/FBt3/detection_loss‚2.667936769398776 trainer/global_step‚8199 _step‚673 0loss/FRt3/assignment_loss‚2.4791359663009644 1loss/SRqqt2/detection_loss‚1.9023812052607536 /loss/FBt4/detection_loss‚2.6829237987597785 epoch‚6 2loss/SRqqt3/assignment_loss‚1.8435718924287827 -loss/FRt4/detection_loss‚2.82465110920571 0loss/SRqqt4/detection_loss‚1.951775367272661 0loss/FRt2/assignment_loss‚2.4110739628473916 0loss/FBt2/assignment_loss‚0.5396332351228921 .loss/FRt1/detection_loss‚1.448987557987372 .loss/FBt2/detection_loss‚2.556268408894539 1loss/SRqqt4/assignment_loss‚1.405156566226419 1loss/FBt3/assignment_loss‚0.44815055604770104 %loss/total_loss‚42.51657353139704 % _timestamp‚1.7884344888703487e+09 /loss/FRt2/detection_loss‚2.313921124911ÙVÒM7757 .loss/FRt3/detection_loss‚3.574393817782402 _runtime‚6173.574958138 1loss/FBt1/assignment_loss‚0.21614646408310353 0loss/SRqqt3/detection_loss‚2.634940182970416 0loss/FRt1/assignment_loss‚1.3558361719051997 /loss/FRt4/assignment_loss‚2.673002750486941 2loss/SRqqt2/assignment_loss‚1.6291064281016587£6ªè] × .loss/FRt3/detection_loss‚3.574393817782402 _runtime‚6173.574958138 1loss/FBt1/assignment_loss‚0.21614646408310353 0loss/SRqqt3/detection_loss‚2.634940182970416 0loss/FRt1/assignment_loss‚1.3558361719051997 /loss/FRt4/assignment_loss‚2.673002750486941 2loss/SRqqt2/assignment_loss‚1.6291064281016587 2loss/SRqqt1/assignment_loss‚1.1203063832736522 1loss/FBt4/assignment_loss‚0.14057852145439634 1loss/SRqqt1/detection_loss‚1.1924523823666122 /loss/FBt1/detection_loss‚0.5042429083071428 .loss/FBt3/detection_loss‚2.667936769398776 trainer/global_step‚8199 _step‚673 0loss/FRt3/assignment_loss‚2.4791359663009644 1loss/SRqqt2/detection_loss‚1.9023812052607536 /loss/FBt4/detection_loss‚2.6829237987597785 epoch‚6 2loss/SRqqt3/assignment_loss‚1.8435718924287827 -loss/FRt4/detection_loss‚2.82465110920571 0loss/SRqqt4/detection_loss‚1.951775367272661 0loss/FRt2/assignment_loss‚2.4110739628473916 0loss/FBt2/assignment_loss‚0.5396332351228921 .loss/FRt1/detection_loss‚1.448987557987372 .loss/FBt2/detection_loss‚2.556268408894539 1loss/SRqqt4/assignment_loss‚1.405156566226419 1loss/FBt3/assignment_loss‚0.44815055604770104 %loss/total_loss‚42.51657353139704 % _timestamp‚1.7884344888703487e+09 /loss/FRt2/detection_loss‚2.3139211249117757¡ÙÑ)ž‘ ‚(:“ ¿°åÔ׬„Ž gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚107.661' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.532200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æËåù'’ ‚(:œ ¿°åÔᬳ‘ disk.nvme0n1.in‚ 921.09765625) proc.memory.availableMB‚ 1026.02734375 disk.sda.out‚ 245.65625 disk.sda1.in‚0 disk.dm-1.out‚ 245.65625 cpu‚25.3346569942654 disk.sda3.in‚0 disk.rbd3.in‚0) proc.memory.percent‚17.06252098083496) disk./.usagePercent‚73.87666402849196 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0& disk.nvme0n1.out‚36434.76513671875 proc.cpu.threads‚17% disk./.usageGB‚1100.8156814575195 disk.sda2.out‚0 disk.sda3.out‚ 245.65625 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚36434.76513671875 disk.rbd0.in‚0 disk.rbd1.out‚ 3.21484375" proc.memory.rssMB‚ 2795.5234375! disk.rbd2.out‚172.12353515625# disk.dm-0.out‚36434.76513671875 disk.sda.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 921.09765625$ memory_percent‚93.73762607574463 network.sent‚ 1.9048088e+07 network.recv‚ 2.12388e+06 disk.drbd1777.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 921.09765625ºYŒž“ ‚(:“ ΰåÔÝ£æ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.271' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.454199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿‹D.” ‚(:£ ΰåÔ¦Ÿ‡ disk.sda.in‚0 disk.nvme0n1.in‚ 921.09765625 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 921.09765625 disk.sda.out‚ 246.30078125 disk.sda1.out‚0 disk.sda3.out‚ 246.30078125 network.sent‚ 1.9153377e+07$ disk.nvme0n1.out‚36534.591796875" disk.nvme0c0n1.in‚ 921.09765625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd2.out‚172.2255859375 disk.drbd1777.out‚0) proc.memory.availableMB‚ 1026.12890625) proc.memory.percent‚17.06252098083496 disk.dm-1.out‚ 246.30078125 disk.sda3.in‚0! disk.dm-0.out‚36534.591796875" proc.memory.rssMB‚ 2795.5234375 disk.sda2.out‚0 disk.rbd3.out‚0$ memory_percent‚93.73700618743896 proc.cpu.threads‚17 network.recv‚ 2.127012e+06% disk./.usageGB‚1100.8157386779785) disk./.usagePercent‚73.87666786860497 disk.rbd1.in‚0 disk.rbd1.out‚ 3.2470703125& disk.nvme0c0n1.out‚36534.591796875 disk.rbd3.in‚0 cpu‚25.3340221381924569Êy¨2• - _wandbruntime‚6208  _runtime‚6208†âüèÇ– Á & lr-AdamW/pg1‚6.282311524441692e-05 & lr-AdamW/pg2‚6.282311524441692e-05 trainer/global_step‚8249 % _timestamp‚1.7884345234077094e+09 _runtime‚6208.112470492 _step‚674‘¦Æ)Ì— Æ & lr-AdamW/pg2‚6.282311524441692e-05 trainer/global_step‚8249 % _timestamp‚1.7884345234077094e+09 _runtime‚6208.112470492 _step‚674 & lr-AdamW/pg1‚6.282311524441692e-05¢»>5Ö2˜ - _wandbruntime‚6208  _runtime‚6208Góoç'™ áO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6875125911160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 275309076 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109180565 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7239048636928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6901183227904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485751742730752 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119724266 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚466315818634240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478851233868288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109180119 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39330351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884345234084754e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119724266 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478851215865344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69850212 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119724264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155584810 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚8249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0[’À2ì'š æO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39329907 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚522956244363776 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚478851233868288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478851215865344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚522956268303872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6875780348916 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155584810 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚8249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6875125911160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 275309076 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109180565 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7239048636928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6901183227904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485751742730752 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119724266 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚466315818634240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478851233868288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109180119 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39330351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884345234084754e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119724266 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478851215865344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69850212 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119724264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275341572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6876596784088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155603290 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275341572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7239906270208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚6208.773088518 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39334441 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚676 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119738280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚473247319548888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530257825311744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523017895101440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7239746853888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478907568904192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155603290 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚478907568904192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39334882 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6901346293248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109193264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚8249 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109192821 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚466370722764800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚466370705725440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275321506 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6875942346124 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 % _timestamp‚1.7884345240678492e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚485809589576192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119738282 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275341572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6876596784088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155603290 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275341572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7239906270208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚6208.773088518 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39334441 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚676 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119738280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚473247319548888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530257825311744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523017895101440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7239746853888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478907568904192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155603290 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚478907568904192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39334882 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6901346293248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109193264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚8249 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109192821 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚466370722764800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚466370705725440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275321506 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6875942346124 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 % _timestamp‚1.7884345240678492e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚485809589576192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119738282 Bž¥ ‚(:“ û°åÔÞÄÛ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚133.574' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.714800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|ûÄâ'¦ ‚(:œ û°åÔ´»† disk.dm-1.in‚0% disk./.usageGB‚1100.8156852722168 disk.nvme0n1.in‚ 921.09765625 disk.sda3.in‚0" proc.memory.rssMB‚ 2795.5234375) proc.memory.percent‚17.06252098083496 disk.rbd2.in‚0 disk.drbd1777.out‚0 network.sent‚ 1.9269422e+07 network.recv‚ 2.137651e+06$ disk.nvme0n1.out‚36733.138671875 disk.rbd0.in‚0 disk.dm-0.in‚ 921.09765625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 3.31005859375 disk.sda2.in‚0 disk.rbd2.out‚173.8154296875 disk.sda3.out‚ 248.2890625 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda.out‚ 248.2890625 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚93.73722076416016( disk./.usagePercent‚73.8766642844995 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 921.09765625& disk.nvme0c0n1.out‚36733.138671875 disk.sda2.out‚0! disk.dm-0.out‚36733.138671875 disk.dm-1.out‚ 248.2890625& proc.memory.availableMB‚ 1026.09375 cpu‚25.33194253550914 disk.drbd1777.in‚0 disk.rbd3.out‚0x\Þ2§ - _wandbruntime‚6242  _runtime‚6242ç×WǨ Á & lr-AdamW/pg1‚6.136400125535712e-05 & lr-AdamW/pg2‚6.136400125535712e-05 trainer/global_step‚8299 % _timestamp‚1.7884345582878067e+09 _runtime‚6242.992598641 _step‚678²i¨óÌ© Æ trainer/global_step‚8299 % _timestamp‚1.7884345582878067e+09 _runtime‚6242.992598641 _step‚678 & lr-AdamW/pg1‚6.136400125535712e-05 & lr-AdamW/pg2‚6.136400125535712e-05¦…Œ 2ª - _wandbruntime‚6242  _runtime‚6242SÝCç'« áO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7281940168704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109819009 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120425064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174109696 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884345582885947e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488611054844416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533320721415680 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156510862 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚481668867807744 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6942169033728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39560395 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6941512671232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚8299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚679 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7282114278400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156510862 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚526038805187072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6941512671232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276915888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6915961250568 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488611054844416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚481668885810688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156490824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488610380478976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120425066 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156490824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70258612 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533320919465472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6916615688324 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120425064 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚481668885810688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70258614 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39559934 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚526038781246976 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚475976969373448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120425066 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚488610380478976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6942169033728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6915961250568 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488611054844416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚481668885810688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonp6®îô itor.on_train_batch_start/active.small_pool.freed‚ 156490824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488610380478976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120425066 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156490824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70258612 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533320919465472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6916615688324 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120425064 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚481668885810688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70258614 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39559934 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚526038781246976 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚475976969373448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120425066 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚488610380478976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6942169033728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120425064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174109696 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884345582885947e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488611054844416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533320721415680 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156510862 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚481668867807744 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6942169033728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39560395 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6941512671232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚8299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚679 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7282114278400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116§®F¨2­ - _wandbruntime‚6243  _runtime‚6243ªñ&® ëM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚8299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120439080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚481725238849536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚481725220846592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚488667552945664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 % _timestamp‚1.7884345589415042e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276948358 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156529342 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚488668227324416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6942988474880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276968424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚481725220846592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7282804723200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚526100431984640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6917432123496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156509278 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚481725238849536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 _step‚680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276948358 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚469115912253440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156509278 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚476032689938972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚488668227324416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120439082 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109831321 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476033361416296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚526100455924736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276968424 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109831773 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6942988474880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70266780 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156529342 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533383424258560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6916777685532 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70266782 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120439082 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚6243.646779376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6917432123496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156509278 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚481725238849536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 _step‚680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276948358 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚469115912253440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156509278 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚476032689938972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚488668227324416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120439082 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109831321 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476033361416296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚526100455924736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276968424 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109831773 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6942988474880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70266780 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156529342 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533383424258560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6916777685532 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70266782 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120439082 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚6243.646779376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚8299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120439080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚481725238849536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚481725220846592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚488667552945664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 % _timestamp‚1.7884345589415042e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276948358 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156529342 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚488668227324416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6942988474880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276968424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚481725220846592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624¨QnéN2° - _wandbruntime‚6243  _runtime‚6243-mH°Y± Ó _runtime‚6243.648104413 0loss/FRt2/assignment_loss‚2.6154454702680763 1loss/FBt3/assignment_loss‚0.17130428133532405 0loss/FBt4/assignment_loss‚0.3402024209499359 /loss/FRt1/detection_loss‚1.5346282949221546 .loss/FBt4/detection_loss‚2.780466690659523 %loss/total_loss‚42.05089026437301 0loss/FBt2/assignment_loss‚0.5837698929632703 1loss/SRqqt1/detection_loss‚1.0123652611460006 0loss/FRt3/assignment_loss‚3.0095110096429525 0loss/FRt4/assignment_loss‚2.7902597368888133 2loss/SRqqt1/assignment_loss‚1.1391372642812452 1loss/SRqqt4/assignment_loss‚1.400163544113176 .loss/FRt3/detection_loss‚2.648630027708254 /loss/FBt1/detection_loss‚0.7790762004729564 _step‚681 0loss/FRt1/assignment_loss‚1.5756011923839306 1loss/SRqqt3/detection_loss‚2.3702702249799454 0loss/SRqqt4/detection_loss‚1.801572679689056 trainer/global_step‚8299 % _timestamp‚1.7884345589434698e+09 1loss/SRqqt2/assignment_loss‚1.566563708050286 1loss/FBt1/assignment_loss‚0.12376274390477188 .loss/FRt2/detection_loss‚2.448054291307926 /loss/FRt4/detection_loss‚2.1115695384313478 1loss/SRqqt2/detection_loss‚1.6396514961824697 .loss/FBt3/detection_loss‚1.905224038021905 2loss/SRqqt3/assignment_loss‚1.8177205983815448 .loss/FBt2/detection_loss‚3.885939657688141 epoch‚6î1›^² Ø 0loss/FRt4/assignment_loss‚2.7902597368888133 2loss/SRqqt1/assignment_loss‚1.1391372642812452 1loss/SRqqt4/assignment_loss‚1.400163544113176 .loss/FRt3/detection_loss‚2.648630027708254 /loss/FBt1/detection_loss‚0.7790762004729564 _step‚681 0loss/FRt1/assignment_loss‚1.5756011923839306 1loss/SRqqt3/detection_loss‚2.3702702249799454 0loss/SRqqt4/detection_loss‚1.801572679689056 trainer/global_step‚8299 % _timestamp‚1.7884345589434698e+09 1loss/SRqqt2/assignment_loss‚1.566563708050286 1loss/FBt1/assignment_loss‚0.12376274390477188 .loss/FRt2/detection_loss‚2.448054291307926 /loss/FRt4/detection_loss‚2.1115695384313478 1loss/SRqqt2/detection_loss‚1.6396514961824697 .loss/FBt3/detection_loss‚1.905224038021905 2loss/SRqqt3/assignment_loss‚1.8177205983815448 .loss/FBt2/detection_loss‚3.885939657688141 epoch‚6 _runtime‚6243.648104413 0loss/FRt2/assignment_loss‚2.6154454702680763 1loss/FBt3/assignment_loss‚0.17130428133532405 0loss/FBt4/assignment_loss‚0.3402024209499359 /loss/FRt1/detection_loss‚1.5346282949221546 .loss/FBt4/detection_loss‚2.780466690659523 %loss/total_loss‚42.05089026437301 0loss/FBt2/assignment_loss‚0.5837698929632703 1loss/SRqqt1/detection_loss‚1.0123652611460006 0loss/FRt3/assignment_loss‚3.0095110096429525©ÑŠá•.³ ‚(:£ Š±åÔ’¾ disk.sda1.out‚0" disk.nvme0c0n1.in‚ 921.09765625 disk.sda3.out‚ 248.57421875 disk.drbd1777.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 1026.08203125) proc.memory.percent‚17.06254482269287 network.recv‚ 2.141356e+06 disk.sda2.out‚0 disk.drbd1777.out‚0! disk.dm-0.out‚36832.990234375 disk.sda.in‚0 disk.dm-1.out‚ 248.57421875$ memory_percent‚93.73729228973389 disk.nvme0n1.in‚ 921.09765625 disk.sda.out‚ 248.57421875 cpu‚25.331265561851648 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 network.sent‚ 1.9273928e+07( disk./.usagePercent‚73.8766642844995$ disk.nvme0n1.out‚36832.990234375& disk.nvme0c0n1.out‚36832.990234375 disk.rbd0.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17! disk.rbd2.out‚173.96435546875% disk./.usageGB‚1100.8156852722168 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 921.09765625 disk.rbd1.in‚0 disk.rbd1.out‚ 3.373046875 disk.sda1.in‚0# proc.memory.rssMB‚ 2795.52734375¾j¯ß´ ‚(:“ Š±åÔõ⟠gpu.0.gpu‚45.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚158.485' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.697000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚Wçl0 œp“µ ‚(:ˆ ™±åÔ“Ü gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚103.039' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÜ({1¶ ‚(:¦ ™±åÔÒ¾ó) disk./.usagePercent‚73.87642850156045% disk./.usageGB‚1100.8121719360352 disk.sda.out‚ 249.16796875 disk.sda1.out‚0 disk.dm-1.out‚ 249.16796875' disk.nvme0c0n1.out‚36932.1181640625 disk.sda3.out‚ 249.16796875# proc.memory.rssMB‚ 2795.52734375 disk.rbd1.in‚0! disk.rbd2.out‚173.97998046875 disk.rbd3.in‚0$ memory_percent‚93.73819828033447) proc.memory.availableMB‚ 1025.93359375 disk.dm-1.in‚0 network.recv‚ 2.148494e+06 disk.sda1.in‚0% disk.nvme0n1.out‚36932.1181640625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 921.09765625 disk.sda.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0) proc.memory.percent‚17.06254482269287 disk.nvme0n1.in‚ 921.09765625 disk.rbd1.out‚ 3.373046875 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 1.9382984e+07" disk.dm-0.out‚36932.1181640625" disk.nvme0c0n1.in‚ 921.09765625 disk.rbd0.in‚0 cpu‚25.33056065988885z<¡2· - _wandbruntime‚6278  _runtime‚6278¯t&ɸ à ' lr-AdamW/pg1‚5.9915713027034746e-05 ' lr-AdamW/pg2‚5.9915713027034746e-05 trainer/global_step‚8349 % _timestamp‚1.7884345933992965e+09 _runtime‚6278.104057664 _step‚682ë­:Xι È _step‚682 ' lr-AdamW/pg1‚5.9915713027034746e-05 ' lr-AdamW/pg2‚5.9915713027034746e-05 trainer/global_step‚8349 % _timestamp‚1.7884345933992965e+09 _runtime‚6278.104057664ªøÈ\ß2º - _wandbruntime‚6278  _runtime‚6278£ä¥æ'» àO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6982498477056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70667012 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6982498477056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491469018227200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚536446319409152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110456231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚529121318130176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121125864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7325171194368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484486537753088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚471806231690240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39789217 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚536446513264640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7325001278976 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚529121342070272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484486519750144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121125866 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39788755 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚471806214650880 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121125864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157436914 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278562780 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚478763682717972 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70667014 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491469018227200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121125866 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157436914 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788434593400042e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6983154839552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 278542740 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157416876 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491469692592640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484486537753088 trainer/global_step‚8349 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110455767 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ji´ë'¼ åO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _runtime‚6278.104967228 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491469018227200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚536446319409152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110456231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚529121318130176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121125864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7325171194368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484486537753088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚471806231690240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39789217 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚536446513264640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7325001278976 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚529121342070272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484486519750144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121125866 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39788755 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚471806214650880 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121125864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157436914 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278562780 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚478763682717972 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70667014 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491469018227200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121125866 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157436914 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788434593400042e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6983154839552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 278542740 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157416876 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491469692592640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484486537753088 trainer/global_step‚8349 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110455767 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6982498477056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70667012 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6982498477056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491526865072640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6983974280704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121139880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484542872788992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121139882 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39793310 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚8349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6983974280704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚536509012378624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 278595276 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484542890791936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7326019570688 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6957613024940 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚529182992807936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278595276 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110468949 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6983317904896 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110468490 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278575210 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121139880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚529182968867840 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157455394 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚471861118781440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6983317904896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚491526190693888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70675180 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7325849668608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚684 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491526865072640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157435330 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521ǼS«ù¿ ïM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 101942)Ë`‘ü%55872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121139880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484542872788992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121139882 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39793310 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚8349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6983974280704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚536509012378624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 278595276 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484542890791936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7326019570688 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6957613024940 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚529182992807936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278595276 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110468949 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6983317904896 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110468490 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278575210 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121139880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚529182968867840 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157455394 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚471861118781440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6983317904896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚491526190693888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70675180 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7325849668608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚684 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491526865072640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157435330 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484542890791936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70675182 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491526865072640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6983974280704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964¬pä§Æ2À - _wandbruntime‚6278  _runtime‚6278Ø#ZÁ Ô /loss/FRt1/detection_loss‚1.6876373025683724 2loss/SRqqt1/assignment_loss‚0.9944858518961285 2loss/SRqqt2/assignment_loss‚1.2502941810712218 1loss/SRqqt2/detection_loss‚1.6604682889580726 1loss/SRqqt4/detection_loss‚1.9182657897472382 _runtime‚6278.758015659 1loss/FBt2/assignment_loss‚0.17599792136621772 0loss/FBt4/assignment_loss‚0.3316430009614963 .loss/FRt2/detection_loss‚2.604001706838608 1loss/SRqqt3/detection_loss‚2.2839156814983914 %loss/total_loss‚40.69151006503883 epoch‚6 /loss/FRt3/assignment_loss‚3.409918263554573 2loss/SRqqt3/assignment_loss‚1.9844452839876925 1loss/FBt1/assignment_loss‚0.07892716310946948 /loss/FBt3/detection_loss‚2.6793309109551564 /loss/FRt4/assignment_loss‚2.670009721298607 .loss/FRt3/detection_loss‚3.010642775467464 /loss/FBt2/detection_loss‚2.3276266008615494 /loss/FBt4/detection_loss‚2.7811510746295633 /loss/FRt2/assignment_loss‚2.600324074923992 2loss/SRqqt4/assignment_loss‚1.3963272376186069 .loss/FRt4/detection_loss‚2.019174798410766 /loss/FBt1/detection_loss‚0.5807487938831423 trainer/global_step‚8349 0loss/FBt3/assignment_loss‚0.3975385743792036 0loss/FRt1/assignment_loss‚1.0428476917500398 1loss/SRqqt1/detection_loss‚0.8057873753032514 $ _timestamp‚1.788434594053362e+09 _step‚685ü¸+í_ Ù /loss/FRt2/assignment_loss‚2.600324074923992 2loss/SRqqt4/assignment_loss‚1.3963272376186069 .loss/FRt4/detection_loss‚2.019174798410766 /loss/FBt1/detection_loss‚0.5807487938831423 trainer/global_step‚8349 0loss/FBt3/assignment_loss‚0.3975385743792036 0loss/FRt1/assignment_loss‚1.0428476917500398 1loss/SRqqt1/detection_loss‚0.8057873753032514 $ _timestamp‚1.788434594053362e+09 _step‚685 /loss/FRt1/detection_loss‚1.6876373025683724 2loss/SRqqt1/assignment_loss‚0.9944858518961285 2loss/SRqqt2/assignment_loss‚1.2502941810712218 1loss/SRqqt2/detection_loss‚1.6604682889580726 1loss/SRqqt4/detection_loss‚1.9182657897472382 _runtime‚6278.758015659 1loss/FBt2/assignment_loss‚0.17599792136621772 0loss/FBt4/assignment_loss‚0.3316430009614963 .loss/FRt2/detection_loss‚2.604001706838608 1loss/SRqqt3/detection_loss‚2.2839156814983914 %loss/total_loss‚40.69151006503883 epoch‚6 /loss/FRt3/assignment_loss‚3.409918263554573 2loss/SRqqt3/assignment_loss‚1.9844452839876925 1loss/FBt1/assignment_loss‚0.07892716310946948 /loss/FBt3/detection_loss‚2.6793309109551564 /loss/FRt4/assignment_loss‚2.670009721298607 .loss/FRt3/detection_loss‚3.010642775467464 /loss/FBt2/detection_loss‚2.3276266008615494 /loss/FBt4/detection_loss‚2.7811510746295633­ÒÓ|žà ‚(:“ ¨±åÔî¥î gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.773' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.354599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N˜€+3Ä ‚(:¨ ¨±åÔÀ ï network.recv‚ 2.150209e+06% disk./.usageGB‚1100.8129234313965 disk.sda2.out‚0! disk.rbd2.out‚174.04248046875( disk./.usagePercent‚73.8764789350447 disk.sda3.out‚ 250.04296875 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 921.09765625 disk.rbd1.in‚0$ memory_percent‚93.73743534088135# proc.memory.rssMB‚ 2795.52734375) proc.memory.percent‚17.06254482269287 network.sent‚ 1.9386387e+07 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 cpu‚25.32992529790024( disk.nvme0c0n1.out‚37031.91552734375 disk.rbd0.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 921.09765625 disk.rbd1.out‚ 3.373046875 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 921.09765625 disk.rbd3.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 250.04296875 disk.drbd1777.out‚0 disk.sda.out‚ 250.04296875& disk.nvme0n1.out‚37031.91552734375 disk.sda2.in‚0 disk.rbd2.in‚0# disk.dm-0.out‚37031.91552734375 disk.sda.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1026.05859375Vf“Å ‚(:ˆ ·±åÔ±ˆð gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚101.426' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ñƒE-Æ ‚(:¢ ·±åÔ¡‡Ê‘ disk.rbd1.out‚ 3.373046875 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.sda.out‚ 250.703125 disk.nvme0n1.in‚ 921.09765625 network.sent‚ 1.9493493e+07 network.recv‚ 2.154559e+06 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚37130.99365234375$ memory_percent‚93.74327659606934) proc.memory.percent‚17.06254482269287 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0! disk.rbd2.out‚175.56982421875 disk.rbd3.in‚0( proc.memory.availableMB‚ 1025.1015625# proc.memory.rssMB‚ 2795.52734375" disk.nvme0c0n1.in‚ 921.09765625 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 250.703125( disk./.usagePercent‚73.8764781670221% disk./.usageGB‚1100.8129119873047( disk.nvme0c0n1.out‚37130.99365234375 proc.cpu.threads‚17 disk.sda3.out‚ 250.703125 disk.dm-0.in‚ 921.09765625# disk.dm-0.out‚37130.99365234375 disk.dm-1.in‚0 cpu‚25.329216260880656ÌËàÏ2Ç - _wandbruntime‚6313  _runtime‚6313uz£XÈÈ Â % _timestamp‚1.7884346286383824e+09 _runtime‚ 6313.34312695 _step‚686 ' lr-AdamW/pg1‚5.8478552380545904e-05 ' lr-AdamW/pg2‚5.8478552380545904e-05 trainer/global_step‚8399üÿJaÍÉ Ç _runtime‚ 6313.34312695 _step‚686 ' lr-AdamW/pg1‚5.8478552380545904e-05 ' lr-AdamW/pg2‚5.8478552380545904e-05 trainer/global_step‚8399 % _timestamp‚1.7884346286383824e+09®N?¬±2Ê - _wandbruntime‚6313  _runtime‚63138)‘ç'Ë áO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _runtime‚6313.344150318 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111092542 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚532203855013376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474551421178880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71075414 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚481549053108264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7023484282880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7023484282880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚481549724585380 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚494327655975424 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7368270045696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7368110616064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71075412 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158362966 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7024140645376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121826666 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532203878953472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158342928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚474551438218240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚494327655975424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121826666 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚487304171692544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539571965629440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6997631929384 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884346286391878e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121826664 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158362966 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111092103 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚8399 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚487304189695488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121826664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚494328330340864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40017128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚539572148999168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 280189632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ÖŸÔFÌ æO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40017128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚539572148999168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 280189632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _runtime‚6313.344150318 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111092542 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚532203855013376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474551421178880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71075414 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚481549053108264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7023484282880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7023484282880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚481549724585380 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚494327655975424 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7368270045696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7368110616064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71075412 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158362966 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7024140645376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121826666 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532203878953472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158342928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚474551438218240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚494327655975424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121826666 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚487304171692544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539571965629440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6997631929384 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_søQÝg¦tart/segment.all.allocated‚844 % _timestamp‚1.7884346286391878e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121826664 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158362966 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111092103 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚8399 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚487304189695488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121826664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚494328330340864¯…ÍæÔ2Í - _wandbruntime‚6314  _runtime‚6314(ç‡ãð&Î êM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7024960086528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚494385502820864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121840680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40021665 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 174967808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚481604773673788 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158381446 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _runtime‚ 6314.00340131 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚487360542734336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111104822 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474606342348800 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7024303710720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7369143035904 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121840682 _step‚688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280222128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8399 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚487360524731392 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280222128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71083582 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚539634497759232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884346292981656e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚494384828442112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚425 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚423 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚474606325309440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚494385502820864 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7368992008192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158361382 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280202062 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121840680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280202062 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71083580 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚532265505751040 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40021242&Áäõ&Ï ïM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚494385502820864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121840680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40021665 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 174967808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚481604773673788 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158381446 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _runtime‚ 6314.00340131 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚487360542734336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111104822 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474606342348800 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7024303710720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7369143035904 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121840682 _step‚688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280222128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8399 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚487360524731392 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280222128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71083582 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚539634497759232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884346292981656e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚494384828442112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚425 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚423 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚474606325309440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚494385502820864 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7368992008192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158361382 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280202062 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121840680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280202062 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71083580 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚532265505751040 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40021242 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7024960086528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128°zå22Ð - _wandbruntime‚6314  _runtime‚6314[ ÄUÑ Ï 2loss/SRqqt1/assignment_loss‚0.9939918252713387 0loss/FBt4/assignment_loss‚0.1911425490139259 /loss/FRt1/detection_loss‚1.6335418696315693 _step‚689 2loss/SRqqt4/assignment_loss‚1.3564002610376624 .loss/FRt2/detection_loss‚2.307049149274826 /loss/FBt1/detection_loss‚0.7031661695772926 % _timestamp‚1.7884346293002608e+09 _runtime‚ 6314.00490872 0loss/FBt1/assignment_loss‚0.1698635616122317 0loss/SRqqt2/detection_loss‚1.926358301265567 0loss/SRqqt3/detection_loss‚2.569386698305607 %loss/total_loss‚41.26827019903655 trainer/global_step‚8399 /loss/FRt3/assignment_loss‚3.047335288741372 2loss/SRqqt2/assignment_loss‚1.3425873948407232 1loss/FBt2/assignment_loss‚0.30884482523136586 .loss/FRt3/detection_loss‚3.372558908029036 .loss/FRt4/detection_loss‚2.123408772507492 1loss/SRqqt1/detection_loss‚1.0565293069692359 epoch‚6 /loss/FRt4/assignment_loss‚2.811791490535347 0loss/FRt1/assignment_loss‚1.2158320237483298 /loss/FRt2/assignment_loss‚2.881783845207908 2loss/SRqqt3/assignment_loss‚1.7457595253363252 1loss/FBt3/assignment_loss‚0.37104729196289554 -loss/FBt3/detection_loss‚2.63388691842556 /loss/FBt2/detection_loss‚2.0942692210276923 1loss/SRqqt4/detection_loss‚1.6316469741649315 .loss/FBt4/detection_loss‚2.780088027318319²[P€ZÒ Ô /loss/FBt2/detection_loss‚2.0942692210276923 1loss/SRqqt4/detection_loss‚1.6316469741649315 .loss/FBt4/detection_loss‚2.780088027318319 2loss/SRqqt1/assignment_loss‚0.9939918252713387 0loss/FBt4/assignment_loss‚0.1911425490139259 /loss/FRt1/detection_loss‚1.6335418696315693 _step‚689 2loss/SRqqt4/assignment_loss‚1.3564002610376624 .loss/FRt2/detection_loss‚2.307049149274826 /loss/FBt1/detection_loss‚0.7031661695772926 % _timestamp‚1.7884346293002608e+09 _runtime‚ 6314.00490872 0loss/FBt1/assignment_loss‚0.1698635616122317 0loss/SRqqt2/detection_loss‚1.926358301265567 0loss/SRqqt3/detection_loss‚2.569386698305607 %loss/total_loss‚41.26827019903655 trainer/global_step‚8399 /loss/FRt3/assignment_loss‚3.047335288741372 2loss/SRqqt2/assignment_loss‚1.3425873948407232 1loss/FBt2/assignment_loss‚0.30884482523136586 .loss/FRt3/detection_loss‚3.372558908029036 .loss/FRt4/detection_loss‚2.123408772507492 1loss/SRqqt1/detection_loss‚1.0565293069692359 epoch‚6 /loss/FRt4/assignment_loss‚2.811791490535347 0loss/FRt1/assignment_loss‚1.2158320237483298 /loss/FRt2/assignment_loss‚2.881783845207908 2loss/SRqqt3/assignment_loss‚1.7457595253363252 1loss/FBt3/assignment_loss‚0.37104729196289554 -loss/FBt3/detection_loss‚2.63388691842556±È¡Ž’Ó ‚(:‡ ƱåÔÎ…ß gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚148.705' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.741 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lID4Ô ‚(:© ƱåÔǬö disk.rbd2.in‚0( disk./.usagePercent‚73.8764781670221 disk.sda1.out‚0 disk.nvme0n1.in‚ 921.09765625( disk.nvme0c0n1.out‚37131.72021484375) proc.memory.availableMB‚ 1025.70703125 network.sent‚ 1.9497206e+07 disk.sda3.out‚ 251.36328125" disk.nvme0c0n1.in‚ 921.09765625 disk.rbd1.in‚0 disk.dm-0.in‚ 921.09765625 disk.sda.out‚ 251.36328125& disk.nvme0n1.out‚37131.72021484375 disk.sda2.out‚0! disk.rbd2.out‚175.63232421875 disk.rbd3.in‚0 disk.rbd0.out‚0$ memory_percent‚93.73958110809326) proc.memory.percent‚17.06254482269287 network.recv‚ 2.157421e+06 disk.sda3.in‚0# disk.dm-0.out‚37131.72021484375 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 3.373046875% disk./.usageGB‚1100.8129119873047 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 251.36328125# proc.memory.rssMB‚ 2795.52734375 cpu‚25.328561198747373 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0Ú›ožÕ ‚(:“ Õ±åÔÖ¿ä gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚141.741' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.348200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cu+q4Ö ‚(:© Õ±åÔ¸æ™& disk.nvme0n1.out‚37230.94384765625 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 network.recv‚ 2.163011e+06 disk.rbd1.in‚0 disk.dm-0.in‚ 921.12890625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 251.92578125$ memory_percent‚93.74125003814697# proc.memory.rssMB‚ 2795.52734375 proc.cpu.threads‚17 network.sent‚ 1.9605401e+07) disk./.usagePercent‚73.87645819843442 disk.sda1.in‚0 disk.dm-1.out‚ 251.92578125 disk.nvme0n1.in‚ 921.12890625 disk.sda2.in‚0! disk.rbd2.out‚175.64794921875 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 921.12890625) proc.memory.availableMB‚ 1025.43359375) proc.memory.percent‚17.06254482269287 cpu‚25.327909693273423 disk.dm-1.in‚0( disk.nvme0c0n1.out‚37230.94384765625 disk.sda3.out‚ 251.92578125 disk.rbd1.out‚ 3.373046875# disk.dm-0.out‚37230.94384765625 disk.sda.in‚0$ disk./.usageGB‚1100.812614440918~ïÒ“× ‚(:ˆ ä±åÔá¼¥Ž gpu.0.gpu‚40.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.512' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02ßG(Ø ‚(: ä±åԳ̑ disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0% proc.memory.availableMB‚ 1025.6875 disk.sda2.out‚0 disk.dm-1.out‚ 252.48046875) proc.memory.percent‚17.06254482269287) disk./.usagePercent‚73.87645845444196 disk.rbd2.out‚ 175.734375 disk.sda3.out‚ 252.48046875 disk.drbd1777.out‚0 cpu‚25.32728705937246% disk./.usageGB‚1100.8126182556152 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 921.12890625% disk.nvme0n1.out‚37330.4755859375 disk.rbd1.in‚0 network.recv‚ 2.166237e+06 disk.sda.out‚ 252.48046875 disk.rbd0.in‚0 disk.sda3.in‚0" disk.dm-0.out‚37330.4755859375 disk.sda1.out‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 921.12890625 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 3.373046875$ memory_percent‚93.73970031738281' disk.nvme0c0n1.out‚37330.4755859375 disk.nvme0n1.in‚ 921.12890625# proc.memory.rssMB‚ 2795.52734375 network.sent‚ 1.9609886e+07 disk.sda2.in‚0×¥~62Ù - _wandbruntime‚6348  _runtime‚6348°5ÊÇÚ Á trainer/global_step‚8449 % _timestamp‚1.7884346635128276e+09 _runtime‚6348.217555086 _step‚690 & lr-AdamW/pg1‚5.705281881801525e-05 & lr-AdamW/pg2‚5.705281881801525e-05åDsÌÛ Æ & lr-AdamW/pg1‚5.705281881801525e-05 & lr-AdamW/pg2‚5.705281881801525e-05 trainer/global_step‚8449 % _timestamp‚1.7884346635128276e+09 _runtime‚6348.217555086 _step‚690²K<Õé2Ü - _wandbruntime‚6348  _runtime‚6348šÝ àO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122527464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7411204494848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeIß6L#viceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542697596391424 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚8449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497186293723648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40246593 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7039121706548 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122527464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚477296627706880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7411366021632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159268980§¯_së'Þ åO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚484335094975672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚490121841637888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497186293723648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7065126451200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7064470088704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281816484 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159289018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71483812 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40246134 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚6348.218389037 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚490121841637888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122527464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚477296627706880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7411366021632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159268980 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122527464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7411204494848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542697596391424 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚8449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497186293723648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40246593 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7039121706548 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281848980 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚542760118877184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281848980 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281828914 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159307498 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚477351548876800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚6348.872224301 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535348042634240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7412076242944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7039283703756 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281828914 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚484391487018520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40250617 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159307498 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7065289516544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159287434 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884346641670702e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122541482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497244140569088 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 trainer/global_step‚8449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122541480 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111742597 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7065289516544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122541480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159287434 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚497244140569088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40251066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535348066574336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7065289516544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122541480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159287434 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚497244140569088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40251066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535348066574336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281848980 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚542760118877184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281848980 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281828914 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159307498 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚477351548876800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚6348.872224301 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535348042634240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7412076242944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7039283703756 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281828914 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚484391487018520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40250617 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159307498 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7065289516544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159287434 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884346641670702e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122541482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497244140569088 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 trainer/global_step‚8449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122541480 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111742597 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928´{ݱ¯2â - _wandbruntime‚6348  _runtime‚6348Qß™ƒ_ã Ù 2loss/FBt4/assignment_loss‚0.059683822037186474 .loss/FRt2/detection_loss‚2.077321475949781 2loss/SRqqt1/assignment_loss‚1.0708793342551763 2loss/SRqqt4/assignment_loss‚1.4495064975624834 1loss/FBt1/assignment_loss‚0.23204571274661678 2loss/FBt3/assignment_loss‚0.012239088703332917 /loss/FRt3/detection_loss‚3.1424052889148393 .loss/FRt4/detection_loss‚2.805299660563469 1loss/SRqqt1/detection_loss‚1.0808676645871909 1loss/SRqqt3/detection_loss‚2.4532531537115574 /loss/FBt1/detection_loss‚0.7071783828956417 _step‚693 0loss/FRt2/assignment_loss‚2.3420101023953537 1loss/FBt2/assignment_loss‚0.03366359086794427 .loss/FBt2/detection_loss‚2.999681830406189 .loss/FBt3/detection_loss‚3.081599678312029 % _timestamp‚1.7884346641686018e+09 epoch‚6 1loss/SRqqt2/detection_loss‚1.7136368541899374 1loss/SRqqt4/detection_loss‚1.7653083591549485 &loss/total_loss‚41.968189472045836 2loss/SRqqt3/assignment_loss‚1.8215551197063178 /loss/FRt1/detection_loss‚1.4029823153935097 /loss/FBt4/detection_loss‚3.3428677022457123 trainer/global_step‚8449 0loss/FRt1/assignment_loss‚1.3437841291780825 /loss/FRt3/assignment_loss‚2.937802647550901 /loss/FRt4/assignment_loss‚2.807723808288574 1loss/SRqqt2/assignment_loss‚1.284893252429063 _runtime‚6348.873223332Øy‘dä Þ 2loss/SRqqt3/assignment_loss‚1.8215551197063178 /loss/FRt1/detection_loss‚1.4029823153935097 /loss/FBt4/detection_loss‚3.3428677022457123 trainer/global_step‚8449 0loss/FRt1/assignment_loss‚1.3437841291780825 /loss/FRt3/assignment_loss‚2.937802647550901 /loss/FRt4/assignment_loss‚2.807723808288574 1loss/SRqqt2/assignment_loss‚1.284893252429063 _runtime‚6348.873223332 2loss/FBt4/assignment_loss‚0.059683822037186474 .loss/FRt2/detection_loss‚2.077321475949781 2loss/SRqqt1/assignment_loss‚1.0708793342551763 2loss/SRqqt4/assignment_loss‚1.4495064975624834 1loss/FBt1/assignment_loss‚0.23204571274661678 2loss/FBt3/assignment_loss‚0.012239088703332917 /loss/FRt3/detection_loss‚3.1424052889148393 .loss/FRt4/detection_loss‚2.805299660563469 1loss/SRqqt1/detection_loss‚1.0808676645871909 1loss/SRqqt3/detection_loss‚2.4532531537115574 /loss/FBt1/detection_loss‚0.7071783828956417 _step‚693 0loss/FRt2/assignment_loss‚2.3420101023953537 1loss/FBt2/assignment_loss‚0.03366359086794427 .loss/FBt2/detection_loss‚2.999681830406189 .loss/FBt3/detection_loss‚3.081599678312029 % _timestamp‚1.7884346641686018e+09 epoch‚6 1loss/SRqqt2/detection_loss‚1.7136368541899374 1loss/SRqqt4/detection_loss‚1.7653083591549485 &loss/total_loss‚41.968189472045836µ5¤+ÀŒå ‚(: ó±åÔª´ò gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚87.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.4 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0våA³'æ ‚(:œ ó±åÔ—¬ ‘ disk.sda2.in‚0" proc.memory.rssMB‚ 2800.9296875 disk.rbd3.in‚0 disk.rbd1.out‚ 3.373046875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 252.98828125 disk.dm-1.in‚0 disk.dm-1.out‚ 252.98828125 disk.nvme0n1.in‚ 921.12890625' disk.nvme0c0n1.out‚37429.6357421875 disk.sda3.out‚ 252.98828125 disk.rbd3.out‚0" disk.dm-0.out‚37429.6357421875 disk.rbd2.in‚0' proc.memory.availableMB‚ 1020.015625 disk.rbd0.in‚0 disk.rbd2.out‚175.75 network.sent‚ 1.9615235e+07 disk.drbd1777.out‚0" disk.nvme0c0n1.in‚ 921.12890625 disk.dm-0.in‚ 921.12890625* proc.memory.percent‚17.095518112182617 proc.cpu.threads‚17 disk.sda1.in‚0) disk./.usagePercent‚73.87645845444196$ memory_percent‚93.77431869506836 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0 cpu‚25.328043933034003 network.recv‚ 2.169389e+06% disk./.usageGB‚1100.8126182556152 disk.drbd1777.in‚0% disk.nvme0n1.out‚37429.6357421875µ²ƒç ‚(:„ ‚²åÔêÇŽ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚86.96' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.392 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M?L¶è ‚(:“ ‚²åÔö¤¤ disk.rbd3.in‚0 disk.rbd0.in‚0 cpu‚25.339781246876996 network.recv‚ 2.17465e+06 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0% proc.memory.availableMB‚ 929.96875# proc.memory.rssMB‚ 2891.24609375) disk./.usagePercent‚73.87645845444196 disk.rbd1.out‚ 3.373046875 disk.sda3.in‚0 disk.sda3.out‚ 253.41796875 network.sent‚ 1.9722158e+07 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 921.12890625% disk./.usageGB‚1100.8126182556152 disk.drbd1777.out‚0 disk.dm-0.in‚ 921.12890625 disk.sda2.out‚0 disk.dm-0.out‚ 37429.984375 disk.dm-1.out‚ 253.41796875! disk.nvme0n1.out‚ 37429.984375 disk.rbd0.out‚0 disk.rbd2.in‚0) proc.memory.percent‚17.64676570892334 disk.sda.out‚ 253.41796875 disk.sda1.in‚0# disk.nvme0c0n1.out‚ 37429.984375 disk.rbd2.out‚ 177.32421875$ memory_percent‚94.32392120361328 disk.rbd3.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 921.12890625 proc.cpu.threads‚17‚¢Ò€é ‚(:‚ ‘²åÔ¶±ê gpu.0.gpu‚52.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚87.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.4 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔìÑ"ê ‚(:— ‘²åÔ¹‚©‘ disk.dm-1.in‚0* proc.memory.percent‚17.647218704223633 proc.cpu.threads‚17 cpu‚25.351174500067017 disk.sda2.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚73.87645845444196% disk./.usageGB‚1100.8126182556152 disk.sda3.out‚ 254.140625 disk.drbd1777.out‚0 disk.sda.out‚ 254.140625# disk.nvme0n1.out‚37529.06640625% disk.nvme0c0n1.out‚37529.06640625 network.recv‚ 2.179243e+06 disk.dm-0.in‚ 921.12890625 disk.dm-0.out‚37529.06640625 disk.nvme0n1.in‚ 921.12890625 disk.rbd2.in‚0$ memory_percent‚94.32556629180908" proc.memory.rssMB‚ 2891.3203125 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.373046875" disk.nvme0c0n1.in‚ 921.12890625 network.sent‚ 1.9728401e+07 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 254.140625 disk.rbd2.out‚ 177.33984375( proc.memory.availableMB‚ 929.69921875 disk.sda2.in‚0ÌM2ë - _wandbruntime‚6404  _runtime‚6404t埶M*ì ÇT D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚492715455592960 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚40531059 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚499809961000960 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚538005437105664 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7067894840304 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 159904707 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 7094505408000 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚71916862 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚499809961000960 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 283141912 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚158 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7441512334336 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 7094505408000 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 283141912 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112447919 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚479794679749632 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚694 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚71916860 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚492715455592960 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 159888730 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112448077 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 159888730 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚486863082543688 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 7068385754696 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚545446949440000 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 7441396990464 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 159904707 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚492715437590016 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 123253182 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 7094012577792 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 123253182 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚538005413165568 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 123253184 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚492715437590016 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚499809450167808 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 _runtime‚6404.853764545ñÔ½¯í ÌT GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚71916862 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚499809961000960 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 283141912 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚158 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7441512334336 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 7094505408000 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 283141912 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112447919 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚479794679749632 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚694 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚71916860 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚492715455592960 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 159888730 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112448077 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 159888730 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚486863082543688 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 7068385754696 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚545446949440000 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 7441396990464 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 159904707 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚492715437590016 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 123253182 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 7094012577792 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 123253182 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚538005413165568 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 123253184 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚492715437590016 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚499809450167808 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 _runtime‚6404.853764545 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚492715455592960 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚40531059 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚499809961000960 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚538005437105664 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7067894840304 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 159904707 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 7094505408000 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0¶°ò Ý’î ‚(:‡  ²åÔƒß gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚170.686' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰t(S.ï ‚(:£  ²åÔ¬ºŒ cpu‚25.362144467141977" proc.memory.rssMB‚ 2891.3203125 disk.sda1.out‚0 disk.nvme0n1.in‚ 921.12890625 disk.sda2.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.rbd2.out‚ 177.46484375 disk.drbd1777.out‚0% disk./.usageGB‚1100.8126754760742 disk.dm-0.in‚ 921.12890625" disk.dm-0.out‚37628.2216796875 disk.rbd1.in‚0( proc.memory.availableMB‚ 929.62109375 network.sent‚ 1.9731844e+07 network.recv‚ 2.180998e+06) disk./.usagePercent‚73.87646229455498 disk.dm-1.in‚0 disk.rbd1.out‚ 3.373046875 disk.sda3.out‚ 254.79296875* proc.memory.percent‚17.647218704223633 proc.cpu.threads‚17 disk.sda.out‚ 254.79296875% disk.nvme0n1.out‚37628.2216796875 disk.dm-1.out‚ 254.79296875 disk.drbd1777.in‚0$ memory_percent‚94.32604312896729 disk.sda.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 921.12890625' disk.nvme0c0n1.out‚37628.2216796875 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 Ê,,2ð - _wandbruntime‚6405  _runtime‚6405*‘H+X)ñ ÒR VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 159912009 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7094870534656 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 283160020 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚545485051016192 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚492752267618816 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚156 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 7068258449712 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚486898739912584 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚479829973509120 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 283160020 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 159912009 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚492752285621760 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚40535669 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚158 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚479829990548480 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚492752285621760 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7094377704448 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 112459191 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚538043312839168 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚40535669 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚158 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚479829990548480 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605·´Ä2ó - _wandbruntime‚6417  _runtime‚6417åd~€Lô Õ§ 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17808930575847626 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5101096630096436 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5014409422874451 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 trainer/global_step‚8462 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9897804856300354 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12757202982902527 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15632183849811554 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3619929552078247 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5032325387001038 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16731734573841095 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5399575233459473 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30767929553985596 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5586028695106506 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2772325575351715 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3209821581840515 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 particle/accuracy_1_of_7‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3833383023738861 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9782149195671082 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5169190764427185 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2708333432674408 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9990444779396057 , Purity/1FRt1SRqqt2FBt/FBt_purity‚0.96875 particle/accuracy_2_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.008641975000500679 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.493055522441864 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_2‚0.7646377682685852 particle/accuracy_4_of_5‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4119528532028198 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚6417.669950511 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5291256308555603 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5416666865348816 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4799351096153259 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3548054099082947 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5623493790626526 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04114546626806259 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1725073903799057 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9753968119621277 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_1_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17477808892726898 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 (particle/f_score‚0.37991735339164734 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/2FRt0SRqqt2FBt/ebddDwvent_purity‚0.04605263099074364 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973471164703369 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16646499931812286 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2542811632156372 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.22860360145568848 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 jet/accuracy_2_of_7‚0.5 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9558307528495789 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2237401157617569 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5625711679458618 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14139047265052795 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09631147980690002 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209877610206604 jet/accuracy_5_of_6‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 particle/accuracy_5_of_7‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9872755408287048 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3875969350337982 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996336042881012 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23759303987026215 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5142350196838379 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.23688271641731262 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 jet/accuracy_4_of_7‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5223891139030457 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_6_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16787058115005493 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 $ _timestamp‚1.788434732963126e+09 _step‚696 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.981524646282196 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17040196061134338 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5118393898010254 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7424241900444031 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4937233328819275 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026280323043465614 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2668515741825104 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.012096773833036423 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06537074595689774 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952515363693237 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 jet/accuracy_5_of_5‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5004158020019531 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21071062982082367 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.026643117889761925 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20773084461688995 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19956162571907043 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17417803406715393 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9860227704048157 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5202571153640747 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17082227766513824 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5555555820465088 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_7‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5912890434265137 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9874313473701477 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.2976655662059784 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.47211945056915283 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.44288504123687744 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11310164630413055 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.40625 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4418998658657074 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.640350878238678 particle/accuracy_4_of_6‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5142350196838379 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49602457880973816 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14176245033740997 +jet/accuracy_2_of_2‚0.10718566924333572 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.40833330154418945 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_6_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.44020718336105347 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25842922925949097 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.13362573087215424 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.38529324531555176 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.5064102411270142 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3499055802822113 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9710884690284729 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9916178584098816 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.016339870169758797 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10623688250780106 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4424329996109009 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.26923078298568726 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44477713108062744 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25065359473228455 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5572563409805298 particle/accuracy_3_of_6‚1 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.561070442199707 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8142338395118713 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34342044591903687 jet/accuracy_6_of_6‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07908590137958527 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 7"Purity/*FRt1SRqqt0FBt/event_purity‚0.26933553814888 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9594214558601379 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17118562757968903 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4004504382610321 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20257610082626343 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2222222238779068 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 particle/accuracy_4_of_4‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611342906952 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2576048672199249 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3004138469696045 +jet/accuracy_2_of_3‚0.25204595923423767 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1866096407175064 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9909908771514893 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984410405158997 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34756946563720703 ,jet/accuracy_4_of_5‚0.008064515888690948 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4416431486606598 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 'particle/accuracy‚0.848446786403656 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.040540196001529694 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4198782444000244 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9425286650657654 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0035651298239827156 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4135115146636963 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007935569621622562 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 epoch‚6 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16149333119392395 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01928599178791046 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15428774058818817 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6000959873199463 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_2_of_5‚0.4090757668018341 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18628863990306854 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018140587955713272 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.48907122015953064 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.49082881212234497 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.641737163066864 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5314149856567383 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8717594742774963 particle/accuracy_1_of_2‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,jet/accuracy_4_of_4‚0.008181705139577389 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9883387684822083 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3997997045516968 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.021677091717720032 particle/accuracy_1_of_6‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5881496667861938 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2250962108373642 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984410405158997 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5618842840194702 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1488063782453537 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5203199982643127 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5522052645683289 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5253633856773376 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.527007520198822 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4252723753452301 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17255645990371704 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5293331742286682 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16518543660640717 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.612500011920929 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05305829644203186 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9898936152458191 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 ,jet/accuracy_4_of_6‚0.057692307978868484 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 )loss/val_total_loss‚42.39174937671406 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.550071120262146 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5502873659133911 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9294373989105225 particle/accuracy_3_of_5‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2277001589536667 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9957525134086609 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.563507616519928 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6027472019195557 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3212963044643402 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9861168265342712 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.961403489112854 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16731734573841095 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.158011794090271 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40076905488967896 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11921240389347076 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.49223265051841736 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9884910583496094 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1879246085882187 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21969695389270782 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24259261786937714 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.44237473607063293 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21001502871513367 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5601043105125427 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9930811524391174 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6152777075767517 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5586481094360352 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3445475101470947 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07131489366292953 particle/accuracy_1_of_3‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6883646249771118 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.491573303937912 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9859883189201355 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5155498385429382 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07555017620325089 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.18666669726371765 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18591950833797455 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2198171317577362 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15506140887737274 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2222222238779068 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26832813024520874 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9503443837165833 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1712237447500229 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012285531498491764 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05119047686457634 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5347222089767456 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.016339870169758797 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9808977842330933 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_2_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5573849678039551 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4830586016178131 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3955357074737549 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17095980048179626 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4958977699279785 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49439650774002075 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.42543864250183105 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_3‚0.026534315198659897 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010997732169926167 +jet/accuracy_1_of_1‚0.48873868584632874 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4099520444869995 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9809197783470154 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03170286864042282 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6132619380950928 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9811331629753113 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1960696280002594 particle/accuracy_1_of_5‚1ÿÛ1bõ Ú§ @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5101096630096436 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5014409422874451 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 trainer/global_step‚8462 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9897804856300354 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12757202982902527 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15632183849811554 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3619929552078247 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5032325387001038 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16731734573841095 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190øüšù531626344 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5399575233459473 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30767929553985596 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5586028695106506 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2772325575351715 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3209821581840515 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 particle/accuracy_1_of_7‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3833383023738861 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9782149195671082 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5169190764427185 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2708333432674408 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9990444779396057 , Purity/1FRt1SRqqt2FBt/FBt_purity‚0.96875 particle/accuracy_2_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.008641975000500679 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.493055522441864 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_2‚0.7646377682685852 particle/accuracy_4_of_5‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4119528532028198 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚6417.669950511 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5291256308555603 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5416666865348816 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4799351096153259 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3548054099082947 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5623493790626526 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04114546626806259 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1725073903799057 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9753968119621277 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_1_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17477808892726898 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 (particle/f_score‚0.37991735339164734 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.04605263099074364 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973471164703369 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16646499931812286 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2542811632156372 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.22860360145568848 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 jet/accuracy_2_of_7‚0.5 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9558307528495789 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2237401157617569 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5625711679458618 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14139047265052795 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09631147980690002 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209877610206604 jet/accuracy_5_of_6‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 particle/accuracy_5_of_7‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9872755408287048 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3875969350337982 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996336042881012 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23759303987026215 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5142350196838379 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.23688271641731262 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 jet/accuracy_4_of_7‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5223891139030457 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_6_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16787058115005493 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 $ _timestamp‚1.788434732963126e+09 _step‚696 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.981524646282196 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17040196061134338 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5118393898010254 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7424241900444031 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4937233328819275 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026280323043465614 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2668515741825104 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.012096773833036423 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06537074595689774 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952515363693237 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 jet/accuracy_5_of_5‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5004158020019531 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21071062982082367 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.026643117889761925 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20773084461688995 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19956162571907043 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17417803406715393 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9860227704048157 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5202571153640747 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17082227766513824 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5555555820465088 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_7‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5912890434265137 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9874313473701477 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.2976655662059784 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.47211945056915283 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.44288504123687744 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11310164630413055 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.40625 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4418998658657074 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.640350878238678 particle/accuracy_4_of_6‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5142350196838379 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49602457880973816 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14176245033740997 +jet/accuracy_2_of_2‚0.10718566924333572 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.40833330154418945 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_6_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.44020718336105347 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25842922925949097 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.13362573087215424 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.38529324531555176 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.5064102411270142 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3499055802822113 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9710884690284729 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9916178584098816 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.016339870169758797 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10623688250780106 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4424329996109009 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.26923078298568726 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44477713108062744 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25065359473228455 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5572563409805298 particle/accuracy_3_of_6‚1 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.561070442199707 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8142338395118713 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34342044591903687 jet/accuracy_6_of_6‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07908590137958527 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 7"Purity/*FRt1SRqqt0FBt/event_purity‚0.26933553814888 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9594214558601379 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17118562757968903 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4004504382610321 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20257610082626343 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2222222238779068 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 particle/accuracy_4_of_4‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611342906952 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2576048672199249 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3004138469696045 +jet/accuracy_2_of_3‚0.25204595923423767 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1866096407175064 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9909908771514893 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984410405158997 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34756946563720703 ,jet/accuracy_4_of_5‚0.008064515888690948 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4416431486606598 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 'particle/accuracy‚0.848446786403656 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.040540196001529694 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4198782444000244 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9425286650657654 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0035651298239827156 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4135115146636963 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007935569621622562 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 epoch‚6 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16149333119392395 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01928599178791046 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15428774058818817 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6000959873199463 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_2_of_5‚0.4090757668018341 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18628863990306854 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018140587955713272 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.48907122015953064 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.49082881212234497 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.641737163066864 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5314149856567383 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8717594742774963 particle/accuracy_1_of_2‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,jet/accuracy_4_of_4‚0.008181705139577389 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9883387684822083 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3997997045516968 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.021677091717720032 particle/accuracy_1_of_6‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5881496667861938 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2250962108373642 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984410405158997 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5618842840194702 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1488063782453537 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5203199982643127 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5522052645683289 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5253633856773376 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.527007520198822 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4252723753452301 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17255645990371704 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5293331742286682 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16518543660640717 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.612500011920929 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05305829644203186 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9898936152458191 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 ,jet/accuracy_4_of_6‚0.057692307978868484 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 )loss/val_total_loss‚42.39174937671406 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.550071120262146 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5502873659133911 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9294373989105225 particle/accuracy_3_of_5‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2277001589536667 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9957525134086609 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.563507616519928 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6027472019195557 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3212963044643402 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9861168265342712 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.961403489112854 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16731734573841095 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.158011794090271 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40076905488967896 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11921240389347076 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.49223265051841736 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9884910583496094 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1879246085882187 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21969695389270782 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24259261786937714 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.44237473607063293 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21001502871513367 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5601043105125427 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9930811524391174 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6152777075767517 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5586481094360352 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3445475101470947 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07131489366292953 particle/accuracy_1_of_3‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6883646249771118 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.491573303937912 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9859883189201355 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5155498385429382 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07555017620325089 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.18666669726371765 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18591950833797455 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2198171317577362 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15506140887737274 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2222222238779068 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26832813024520874 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9503443837165833 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1712237447500229 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012285531498491764 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05119047686457634 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5347222089767456 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.016339870169758797 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9808977842330933 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_2_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5573849678039551 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4ûyᆠ830586016178131 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3955357074737549 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17095980048179626 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4958977699279785 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49439650774002075 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.42543864250183105 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_3‚0.026534315198659897 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010997732169926167 +jet/accuracy_1_of_1‚0.48873868584632874 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4099520444869995 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9809197783470154 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03170286864042282 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6132619380950928 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9811331629753113 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1960696280002594 particle/accuracy_1_of_5‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17808930575847626 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1¸›¦kZ›ö ‚(: ¯²åÔû“›Ž gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚30.0 gpu.0.powerWatts‚76.02' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.203999999999999 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;:I'÷ ‚(:œ ¯²åÔ¶¢Ô‘ disk.drbd1777.out‚0) disk./.usagePercent‚73.87659157835978" disk.nvme0c0n1.in‚ 921.12890625 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda1.out‚0 network.sent‚ 1.9827917e+07 network.recv‚ 2.185653e+06 disk.rbd0.out‚0 disk.drbd1777.in‚0! disk.dm-0.out‚37727.986328125& disk.nvme0c0n1.out‚37727.986328125 disk.rbd3.out‚0 disk.sda.out‚ 255.3359375 disk.sda1.in‚0) proc.memory.availableMB‚ 1426.33203125 disk.dm-1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 921.12890625* proc.memory.percent‚17.545604705810547 disk.rbd0.in‚0 disk.rbd2.out‚ 177.4765625 disk.sda3.out‚ 255.3359375 cpu‚25.371333754206173% disk./.usageGB‚1100.8146018981934 disk.dm-1.out‚ 255.3359375 disk.sda2.in‚0! proc.memory.rssMB‚ 2874.671875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 3.373046875$ memory_percent‚91.29436016082764 disk.nvme0n1.in‚ 921.12890625$ disk.nvme0n1.out‚37727.986328125 disk.sda3.in‚0¢šÚ“ø ‚(:ˆ ¾²åÔúñã gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.0442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ª¬[3ù ‚(:¨ ¾²åÔÚô‘& disk.nvme0n1.out‚38300.08935546875" disk.nvme0c0n1.in‚ 921.12890625 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda2.in‚0) disk./.usagePercent‚73.87659541847277 disk.sda2.out‚0 disk.dm-0.in‚ 921.12890625 disk.sda.in‚0 disk.sda.out‚ 256.05859375# disk.dm-0.out‚38300.08935546875 disk.rbd1.out‚ 3.373046875 disk.rbd0.out‚0 disk.rbd2.in‚0) proc.memory.percent‚17.54765510559082 disk.rbd3.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 921.12890625! disk.rbd2.out‚177.56298828125$ memory_percent‚91.29655361175537) proc.memory.availableMB‚ 1425.97265625 network.sent‚ 1.9960789e+07% disk./.usageGB‚1100.8146591186523 disk.drbd1777.in‚0 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚38300.08935546875 cpu‚25.37028350889658 network.recv‚ 2.190131e+06 disk.sda3.in‚0 disk.sda3.out‚ 256.05859375 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2875.0078125 disk.dm-1.out‚ 256.05859375m)_2ú - _wandbruntime‚6447  _runtime‚6447Ÿ*•Äû ¾ & lr-AdamW/pg1‚5.563880946018031e-05 & lr-AdamW/pg2‚5.563880946018031e-05 trainer/global_step‚8499 # _timestamp‚1.78843476278265e+09 _runtime‚ 6447.48738907 _step‚697ß.ßÛÉü à & lr-AdamW/pg1‚5.563880946018031e-05 & lr-AdamW/pg2‚5.563880946018031e-05 trainer/global_step‚8499 # _timestamp‚1.78843476278265e+09 _runtime‚ 6447.48738907 _step‚697¹–è52ý - _wandbruntime‚6447  _runtime‚6447‚Pa£æ'þ àO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7129132227072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚540755099990528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7102881546038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160677033 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884347627835143e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284586077 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123909042 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489367568190906 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚482265358121984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7129132227072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123909044 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123909044 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7477613485568 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7102227108282 trainer/global_step‚8499 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160677033 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502388243256320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 6447.48849876 _step‚698 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482265341082624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40756294 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚548232737416192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚502388917621760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚502388243256320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72301964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113058260 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160656995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489368239668022 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160656995 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7477449861632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0:,™ë'ÿ åO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160656995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489368239668022 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160656995 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7477449861632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7129132227072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚540755099990528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7102881546038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160677033 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884347627835143e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284586077 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123909042 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489367568190906 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚482265358121984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7129132227072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123909044 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123909044 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7477613485568 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7102227108282 trainer/global_step‚8499 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160677033 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502388243256320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 6447.48849876 _step‚698 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482265341082624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40756294 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚548232737416192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚502388917621760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚502388243256320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72301964 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113058260 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944º‰„5Ò2€!- _wandbruntime‚6448  _runtime‚6448i¼Å¿£!ëM trainer/global_step‚8499 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884347634428504e+09 _runtime‚6448.148151029 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123923058 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚540816750728192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123923060 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160675449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72310134 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7478462512128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7103697981210 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚502446090101760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482320245213184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160675449 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7103043543246 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40760296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160695513 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502445415723008 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160695513 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚502446090101760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D6DeviceStatsMonitor.on_train_batch_end/a/•Ì5Nctive.all.freed‚ 284598507 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7129295292416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚489423288756430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚699 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7129951668224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72310132 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284618573 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚495316120430592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489423960233754 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚540816774668288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123923060 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284598507 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113070428 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40760726 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚548295039143936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚495316138433536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548295237180416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284618573 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844¹Ÿ±ö&‚!ðM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160675449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72310134 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7478462512128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7103697981210 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚502446090101760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482320245213184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160675449 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7103043543246 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40760296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160695513 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502445415723008 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160695513 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚502446090101760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284598507 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7129295292416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚489423288756430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚699 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7129951668224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72310132 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284618573 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚495316120430592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489423960233754 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚540816774668288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123923060 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284598507 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113070428 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40760726 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚548295039143936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚495316138433536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548295237180416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284618573 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚8499 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884347634428504e+09 _runtime‚6448.148151029 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123923058 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚540816750728192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123923060 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚543837660813824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505247555369984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7143062447690 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286192889 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124609844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚543837636873728 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚492153610058314 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚551358089937920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505247555369984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7170118032896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚498077437337088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚498077419334144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 $ _timestamp‚1.788434798143092e+09 _runtime‚6482.848001128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124609842 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚551357893985280 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚492154281535430 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161583047 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113690245 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72710364 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚498077419334144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7143716885446 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286212929 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚498077437337088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7520429124096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚505246881004544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40980325 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40979881 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113690691 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7169461670400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 trainer/global_step‚8549 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161603085 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7520257111552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚485010564649984 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505246881004544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7170118032896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚485010547610624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚702 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161583047 Ly¿2‘!åO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286192889 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124609844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚543837636873728 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚492153610058314 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_splvÉè¹%it_bytes.all.allocated‚551358089937920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505247555369984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7170118032896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚498077437337088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚498077419334144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 $ _timestamp‚1.788434798143092e+09 _runtime‚6482.848001128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124609842 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚551357893985280 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚492154281535430 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161583047 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113690245 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72710364 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚498077419334144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7143716885446 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286212929 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚498077437337088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7520429124096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚505246881004544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40980325 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40979881 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113690691 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7169461670400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 trainer/global_step‚8549 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161603085 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7520257111552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚485010564649984 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505246881004544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7170118032896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚485010547610624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚702 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161583047 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124609844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124609842 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72710366 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚543837660813824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505247555369984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7143062447690 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0¾¢U±2’!- _wandbruntime‚6483  _runtime‚6483°Â¥wð&“!êM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161601501 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚505304053471232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286245425 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505304727849984 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124623860 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚485065451741184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124623858 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161601501 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚543899287611392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7170281098240 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7170281098240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161621565 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124623860 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7143878882654 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7170937474048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7521286018048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7144533320618 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚498133790375936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40984321 $ _timestamp‚1.788434798802787e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286225359 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7170937474048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚551420597569536 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124623858 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚498133772372992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113703290 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚492209330623838 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113702853 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚543899311551488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286245425 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505304727849984 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124623860 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚485065451741184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124623858 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161601501 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚543899287611392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7170281098240 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7170281098240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161621565 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124623860 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7143878882654 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7170937474048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7521286018048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7144533320618 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚498133790375936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40984321 $ _timestamp‚1.788434798802787e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286225359 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7170937474048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚551420597569536 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124623858 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚498133772372992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113703290 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚492209330623838 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113702853 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚543899311551488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73118766 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚554483480862208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125310642 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7211103838720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚546920173756928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884348331032147e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚508106193118208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287839781 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7183897787098 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7211103838720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41204237 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508105518752768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7210447476224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125310642 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114323001 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚494940323402838 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7184552224854 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162509099 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162509099 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚554483676814848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚487755754138624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7563307105280 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162529137 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494939651925722 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7210447476224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚508106193118208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73118764 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487755771177984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125310644 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚500895089279488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7563479117824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚706 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287819741 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚500895071276544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162529137 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚8599 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114323454 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546920197697024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41204688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚6517.808109928««††ì'¡!æO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7210447476224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚508106193118208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73118764 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487755771177984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125310644 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚500895089279488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7563479117824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚706 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287819741 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚500895071276544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162529137 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚8599 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114323454 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546920197697024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41204688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚6517.808109928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚500895089279488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚500895071276544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73118766 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚554483480862208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125310642 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7211103838720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚546920173756928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884348331032147e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚508106193118208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287839781 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7183897787098 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7211103838720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41204237 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508105518752768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7210447476224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125310642 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114323001 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚494940323402838 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7184552224854 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162509099 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162509099 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚554483676814848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚487755754138624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7563307105280 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162529137 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494939651925722 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚707 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114336036 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162547617 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287852211 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114335602 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚546981824494592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚546981848434688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7564186619392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚487810658269184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7184714222062 trainer/global_step‚8599 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508162691219456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7211923279872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125324660 _runtime‚6518.476144402 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7211266904064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7185368660026 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚508163365598208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚494996043968570 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚500951424315392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚434 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73126934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41209102 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287872277 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚494995372491246 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125324658 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚500951424315392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7211923279872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41208670 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162527553 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚554546188178944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287852211 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚508163365598208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162547617 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7211266904064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.o‰(‡ö¥n_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125324660 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7564339744256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376…¢‚%ö&¤!ðM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287872277 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚494995372491246 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125324658 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚500951424315392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7211923279872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41208670 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162527553 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚554546188178944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287852211 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚508163365598208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162547617 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7211266904064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125324660 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7564339744256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚707 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114336036 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162547617 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287852211 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114335602 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚546981824494592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚546981848434688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7564186619392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚487810658269184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7184714222062 trainer/global_step‚8599 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508162691219456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7211923279872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125324660 _runtime‚6518.476144402 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7211266904064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7185368660026 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚508163365598208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚494996043968570 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚500951424315392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚434 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73126934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41209102Ã9Þ.2¥!- _wandbruntime‚6518  _runtime‚6518¤ãK³X¦!Ò epoch‚7 2loss/SRqqt2/assignment_loss‚1.1620564638920452 1loss/SRqqt4/assignment_loss‚1.605607708945634 /loss/FRt3/detection_loss‚3.3249218871718957 0loss/FRt3/assignment_loss‚2.7048091449235616 1loss/FBt1/assignment_loss‚0.16296856630293696 /loss/FRt2/detection_loss‚2.5677444065610566 1loss/SRqqt4/detection_loss‚1.7263970573743184 .loss/FBt3/detection_loss‚2.672028402487437 _step‚708 0loss/FRt4/assignment_loss‚2.7989585507999766 /loss/FRt1/detection_loss‚1.6718802032327014 /loss/FBt1/detection_loss‚0.5145466087420786 /loss/FBt4/detection_loss‚2.4390977025032043 _runtime‚6518.477220349 1loss/SRqqt3/assignment_loss‚1.560343866981566 0loss/FBt3/assignment_loss‚0.2585390778258443 0loss/SRqqt1/detection_loss‚1.131092021415631 trainer/global_step‚8599 0loss/FRt1/assignment_loss‚1.4693145081400871 /loss/FRt2/assignment_loss‚2.889529373910692 0loss/FBt2/assignment_loss‚0.2659061671414141 0loss/FBt4/assignment_loss‚0.4825330413877964 .loss/FRt4/detection_loss‚2.359641081230207 0loss/SRqqt2/detection_loss‚2.004766414994779 .loss/FBt2/detection_loss‚2.975327501694361 2loss/SRqqt1/assignment_loss‚0.9886510830869277 % _timestamp‚1.7884348337725892e+09 1loss/SRqqt3/detection_loss‚2.1148504972457887 %loss/total_loss‚41.85151133799194JÞ‰]§!× 0loss/FBt3/assignment_loss‚0.2585390778258443 0loss/SRqqt1/detection_loss‚1.131092021415631 trainer/global_step‚8599 0loss/FRt1/assignment_loss‚1.4693145081400871 /loss/FRt2/assignment_loss‚2.889529373910692 0loss/FBt2/assignment_loss‚0.2659061671414141 0loss/FBt4/assignment_loss‚0.4825330413877964 .loss/FRt4/detection_loss‚2.359641081230207 0loss/SRqqt2/detection_loss‚2.004766414994779 .loss/FBt2/detection_loss‚2.975327501694361 2loss/SRqqt1/assignment_loss‚0.9886510830869277 % _timestamp‚1.7884348337725892e+09 1loss/SRqqt3/detection_loss‚2.1148504972457887 %loss/total_loss‚41.85151133799194 epoch‚7 2loss/SRqqt2/assignment_loss‚1.1620564638920452 1loss/SRqqt4/assignment_loss‚1.605607708945634 /loss/FRt3/detection_loss‚3.3249218871718957 0loss/FRt3/assignment_loss‚2.7048091449235616 1loss/FBt1/assignment_loss‚0.16296856630293696 /loss/FRt2/detection_loss‚2.5677444065610566 1loss/SRqqt4/detection_loss‚1.7263970573743184 .loss/FBt3/detection_loss‚2.672028402487437 _step‚708 0loss/FRt4/assignment_loss‚2.7989585507999766 /loss/FRt1/detection_loss‚1.6718802032327014 /loss/FBt1/detection_loss‚0.5145466087420786 /loss/FBt4/detection_loss‚2.4390977025032043 _runtime‚6518.477220349 1loss/SRqqt3/assignment_loss‚1.560343866981566Ä{üBž¨!‚(:“ ˜³åÔ±­ê gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.337' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.267399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÆœ,.©!‚(:£ ˜³åÔ€å¡ disk.sda2.out‚0$ disk./.usageGB‚1100.812931060791 disk.sda3.in‚0 disk.dm-1.out‚ 259.43359375" proc.memory.rssMB‚ 2875.0078125 proc.cpu.threads‚17 disk.rbd3.out‚0) proc.memory.availableMB‚ 1426.05078125 disk.rbd2.in‚0 disk.sda3.out‚ 259.43359375" disk.dm-0.out‚39171.4248046875 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 921.12890625$ memory_percent‚91.29607677459717) proc.memory.percent‚17.54765510559082 disk.sda2.in‚0 network.recv‚ 2.210411e+06 disk.rbd2.out‚179.4580078125 disk.drbd1777.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 921.12890625 disk.rbd0.in‚0 network.sent‚ 2.0191127e+07 disk.drbd1777.out‚0 disk.sda.out‚ 259.43359375 disk.rbd0.out‚0 disk.dm-1.in‚0 cpu‚25.36577485621596) disk./.usagePercent‚73.87647944705978 disk.rbd3.in‚0 disk.dm-0.in‚ 921.12890625% disk.nvme0n1.out‚39171.4248046875 disk.rbd1.out‚ 3.373046875' disk.nvme0c0n1.out‚39171.4248046875²ìËD“ª!‚(:ˆ §³åÔý´Ò gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g°B «!‚(:• §³åÔõ€Þ network.recv‚ 2.215464e+06% disk./.usageGB‚1100.8145179748535 disk.rbd2.out‚180.9853515625 disk.sda3.out‚ 259.9296875 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 921.12890625 disk.dm-0.in‚ 921.12890625 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚91.29600524902344% proc.memory.availableMB‚ 1426.0625 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda1.out‚0" disk.nvme0n1.out‚ 39272.4140625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.373046875 disk.dm-1.out‚ 259.9296875 disk.sda2.in‚0) disk./.usagePercent‚73.87658594619401 disk.dm-0.out‚ 39272.4140625 disk.dm-1.in‚0$ disk.nvme0c0n1.out‚ 39272.4140625 cpu‚25.365049768963548 network.sent‚ 2.0299042e+07 disk.sda.out‚ 259.9296875 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 921.12890625" proc.memory.rssMB‚ 2875.0078125) proc.memory.percent‚17.54765510559082«Äî2¬!- _wandbruntime‚6552  _runtime‚65527,ILÉ­!à ' lr-AdamW/pg2‚5.1470060804684035e-05 trainer/global_step‚8649 % _timestamp‚1.7884348681620297e+09 _runtime‚6552.866808786 _step‚709 ' lr-AdamW/pg1‚5.1470060804684035e-05ô1K’ή!È ' lr-AdamW/pg1‚5.1470060804684035e-05 ' lr-AdamW/pg2‚5.1470060804684035e-05 trainer/global_step‚8649 % _timestamp‚1.7884348681620297e+09 _runtime‚6552.866808786 _step‚709Åßc]¤2¯!- _wandbruntime‚6552  _runtime‚6552BZ‹Pæ'°!àO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289466633 _runtime‚6552.867715434 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7252089644544 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114957916 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41430752 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7251433282048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7252089644544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163455189 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚8649 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚497726365270246 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7251433282048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73527164 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚510964830866432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557609125210112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7224733126506 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126011442 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41431186 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚510964156500992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚503712741221888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126011444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚557609310676992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289446593 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126011444 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚497725693793130 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73527166 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚503712723218944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚503712741221888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 $ _timestamp‚1.788434868162746e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7606576096768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7606414569984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7225387564262 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚710 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163435151 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510964156500992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚8449`k൱!åO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163435151 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚550002710640128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163455189 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510964830866432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚503712723218944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510964156500992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289466633 _runtime‚6552.867715434 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7252089644544 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114957916 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41430752 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7251433282048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7252089644544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163455189 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚8649 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚497726365270246 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7251433282048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73527164 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚510964830866432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557609125210112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7224733126506 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large²ZCÑ6_pool.freed‚ 126011442 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41431186 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚510964156500992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚503712741221888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126011444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚557609310676992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289446593 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126011444 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚497725693793130 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73527166 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚503712723218944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚503712741221888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 $ _timestamp‚1.788434868162746e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7606576096768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7606414569984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7225387564262 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚710 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163435151 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114970691 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚503769094260736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚511022003346432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126025458 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚550064361377792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚497782085835978 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550064385317888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7607433068544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163473669 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163453605 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚511022003346432 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚497781414358654 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289499129 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7252909085696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚557671618252800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7225549561470 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚503769094260736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289479063 $ _timestamp‚1.788434868827794e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289499129 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41435359 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41435799 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚6553.532928392 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73535334 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7226203999434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0¡‰Áõ&´!ïM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289499129 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7252909085696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚557671618252800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7225549561470 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚503769094260736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289479063 $ _timestamp‚1.788434868827794e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289499129 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41435359 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41435799 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚6553.532928392 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73535334 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7226203999434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163473669 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7252909085696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 trainer/global_step‚8649 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7252252709888 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163453605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚511021328967680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126025460 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126025458 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114970691 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚503769094260736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚511022003346432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126025458 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚550064361377792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚497782085835978 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550064385317888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7607433068544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163473669 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163453605 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚511022003346432 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚497781414358654Ç ¤J2µ!- _wandbruntime‚6553  _runtime‚6553A†#Y¶!Ó 0loss/SRqqt4/assignment_loss‚1.09904527604166 0loss/FBt1/assignment_loss‚0.0674951282898188 /loss/FRt4/detection_loss‚2.3083208578949175 /loss/FBt1/detection_loss‚0.5363935449599384 0loss/FRt3/assignment_loss‚2.9650047479137296 1loss/SRqqt1/detection_loss‚0.9259483419673543 _step‚712 2loss/SRqqt3/assignment_loss‚1.7950093830004334 1loss/SRqqt2/detection_loss‚1.8176523847712411 .loss/FBt2/detection_loss‚2.756107896566391 /loss/FBt4/detection_loss‚2.3518739257540022 _runtime‚6553.534044475 2loss/SRqqt2/assignment_loss‚1.5255260918703344 /loss/FRt3/detection_loss‚2.8029021447704685 /loss/FBt4/assignment_loss‚0.736345376619803 /loss/FRt1/detection_loss‚1.8283237824216485 .loss/FRt2/detection_loss‚2.470676925920305 1loss/SRqqt3/detection_loss‚2.4662731965382894 /loss/FBt3/detection_loss‚2.6108179926872253 %loss/total_loss‚41.20460178836008 2loss/SRqqt1/assignment_loss‚1.0139789120228568 0loss/FBt3/assignment_loss‚0.2339801378948323 0loss/FRt2/assignment_loss‚2.2157236493769146 0loss/FRt4/assignment_loss‚3.0250602811574936 1loss/SRqqt4/detection_loss‚2.2276437358723746 % _timestamp‚1.7884348688294206e+09 /loss/FRt1/assignment_loss‚1.231017634920452 0loss/FBt2/assignment_loss‚0.1934804391275975 epoch‚7 trainer/global_step‚8649šn³ç^·!Ø 0loss/FRt3/assignment_loss‚2.9650047479137296 1loss/SRqqt1/detection_loss‚0.9259483419673543 _step‚712 2loss/SRqqt3/assignment_loss‚1.7950093830004334 1loss/SRqqt2/detection_loss‚1.8176523847712411 .loss/FBt2/detection_loss‚2.756107896566391 /loss/FBt4/detection_loss‚2.3518739257540022 _runtime‚6553.534044475 2loss/SRqqt2/assignment_loss‚1.5255260918703344 /loss/FRt3/detection_loss‚2.8029021447704685 /loss/FBt4/assignment_loss‚0.736345376619803 /loss/FRt1/detection_loss‚1.8283237824216485 .loss/FRt2/detection_loss‚2.470676925920305 1loss/SRqqt3/detection_loss‚2.4662731965382894 /loss/FBt3/detection_loss‚2.6108179926872253 %loss/total_loss‚41.20460178836008 2loss/SRqqt1/assignment_loss‚1.0139789120228568 0loss/FBt3/assignment_loss‚0.2339801378948323 0loss/FRt2/assignment_loss‚2.2157236493769146 0loss/FRt4/assignment_loss‚3.0250602811574936 1loss/SRqqt4/detection_loss‚2.2276437358723746 % _timestamp‚1.7884348688294206e+09 /loss/FRt1/assignment_loss‚1.231017634920452 0loss/FBt2/assignment_loss‚0.1934804391275975 epoch‚7 trainer/global_step‚8649 0loss/SRqqt4/assignment_loss‚1.09904527604166 0loss/FBt1/assignment_loss‚0.0674951282898188 /loss/FRt4/detection_loss‚2.3083208578949175 /loss/FBt1/detection_loss‚0.5363935449599384ÈL¾L¬’¸!‚(:‡ ¶³åÔÇåç gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.623 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XSåÛ2¹!‚(:§ ¶³åÔÉÙÀ disk.drbd1777.out‚0 disk.sda.in‚0 disk.dm-1.in‚0% disk./.usageGB‚1100.8145179748535 disk.rbd1.out‚ 3.373046875( disk.nvme0c0n1.out‚39371.63330078125 disk.rbd2.out‚181.0478515625 disk.sda.out‚ 260.48828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 260.48828125 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 921.12890625 network.recv‚ 2.219268e+06 disk.dm-1.out‚ 260.48828125$ memory_percent‚91.29748344421387( proc.memory.availableMB‚ 1425.8203125 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 921.12890625 disk.drbd1777.in‚0# disk.dm-0.out‚39371.63330078125) proc.memory.percent‚17.54765510559082 cpu‚25.364371400711526) disk./.usagePercent‚73.87658594619401 disk.nvme0n1.in‚ 921.12890625& disk.nvme0n1.out‚39371.63330078125 disk.rbd2.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2875.0078125 network.sent‚ 2.0303642e+07 disk.sda3.in‚0F Ëžº!‚(:“ ųåÔ•†þ gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.304' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.860799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kHǦ.»!‚(:£ ųåÔØ»§‘ disk.drbd1777.out‚0 cpu‚25.363612253359577 network.recv‚ 2.225904e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 261.2265625( disk.nvme0c0n1.out‚39471.27685546875( proc.memory.availableMB‚ 1425.8984375 disk.sda2.out‚0 disk.sda1.in‚0( disk./.usagePercent‚73.8765879942543" disk.nvme0c0n1.in‚ 921.12890625 disk.sda.out‚ 261.2265625 disk.rbd0.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1100.8145484924316 disk.rbd3.out‚0# disk.dm-0.out‚39471.27685546875 disk.sda1.out‚0 disk.nvme0n1.in‚ 921.12890625& disk.nvme0n1.out‚39471.27685546875 disk.rbd1.in‚0 disk.rbd2.out‚181.0712890625 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 921.12890625 disk.sda.in‚0$ memory_percent‚91.29700660705566" proc.memory.rssMB‚ 2875.0078125 disk.rbd1.out‚ 3.373046875 disk.rbd0.in‚0) proc.memory.percent‚17.54765510559082 network.sent‚ 2.0412622e+07 disk.sda3.out‚ 261.2265625 disk.drbd1777.in‚0.{/ ’¼!‚(:‡ Ô³åÔ‚’’Ž gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚88.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çv ¨0½!‚(:¥ Ô³åÔ€Óª& proc.memory.availableMB‚ 1425.65625 cpu‚25.362908374123027) disk./.usagePercent‚73.87658671421661% disk./.usageGB‚1100.8145294189453( disk.nvme0c0n1.out‚39471.56201171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0) proc.memory.percent‚17.54765510559082 disk.sda.in‚0 disk.rbd1.out‚ 3.373046875" disk.nvme0c0n1.in‚ 921.12890625 disk.sda2.in‚0& disk.nvme0n1.out‚39471.56201171875! disk.rbd2.out‚181.22021484375 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 921.12890625 disk.sda2.out‚0 disk.sda3.out‚ 261.69921875# disk.dm-0.out‚39471.56201171875" proc.memory.rssMB‚ 2875.0078125 network.sent‚ 2.0418167e+07 disk.rbd1.in‚0 disk.dm-0.in‚ 921.12890625 disk.sda1.in‚0# memory_percent‚91.2984848022461 network.recv‚ 2.229884e+06 disk.sda3.in‚0 disk.sda.out‚ 261.69921875 disk.dm-1.in‚0 disk.dm-1.out‚ 261.69921875?D%õ2¾!- _wandbruntime‚6587  _runtime‚6587ñ\XJÉ¿!à _step‚713 ' lr-AdamW/pg1‚5.0105869688862534e-05 ' lr-AdamW/pg2‚5.0105869688862534e-05 trainer/global_step‚8699 % _timestamp‚1.7884349029737284e+09 _runtime‚6587.678459435ShyÎÀ!È _step‚713 ' lr-AdamW/pg1‚5.0105869688862534e-05 ' lr-AdamW/pg2‚5.0105869688862534e-05 trainer/global_step‚8699 % _timestamp‚1.7884349029737284e+09 _runtime‚6587.678459435É+ñu32Á!- _wandbruntime‚6587  _runtime‚6587P7¬!ç'Â!áO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164361203 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41657229 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 trainer/global_step‚8699 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291073445 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚506530393164288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚513823468614656 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126712242 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6587.679330205 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚560734503703040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚506530393164288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7292419087872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115592795 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115592339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73935564 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7649413512192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7265568465914 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚506530375161344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164361203 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚493246167194624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚506530375161344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7292419087872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126712242 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚560734684975616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚500512407137654 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7266222903670 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291093485 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513822794249216 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7293075450368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚714 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41656775 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048#=i’ì'Ã!æO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553085247523328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚513823468614656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7292419087872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115592795 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115592339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73935564 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7649413512192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7265568465914 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚506530375161344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164361203 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚493246167194624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚506530375161344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7292419087872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126712242 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚560734684975616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚500512407137654 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7266222903670 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291093485 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513822794249216 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7293075450368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚714 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41656775 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164361203 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41657229 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 trainer/global_step‚8699 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291073445 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚506530393164288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚513823468614656 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126712242 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6587.679330205 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚560734503703040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚506530393164288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291125981 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7293894891520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7650277993984 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚493301088364544 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126726260 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126726258 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚500567456226062 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553146898260992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73943734 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚513880641094656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚560797006352896 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115604919 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚ 6588.34050373 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7650108091904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚506586728200192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7293238515712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115605380 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164379657 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291105915 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291105915 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164399721 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚506586728200192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚500568127703386 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚493301071325184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0«âêèÆ!ïM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164399721 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚506586728200192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚500568127703386 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚493301071325184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291125981 J@DeviceStatsMonitor.on_trainiÌA— _batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7293894891520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7650277993984 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚493301088364544 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126726260 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126726258 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚500567456226062 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553146898260992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73943734 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚513880641094656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚560797006352896 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115604919 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚ 6588.34050373 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7650108091904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚506586728200192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7293238515712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115605380 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164379657 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291105915 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291105915 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064ËsrgJ2Ç!- _wandbruntime‚6588  _runtime‚6588ä’~ÙZÈ!Ô 0loss/FRt1/assignment_loss‚0.7797990043958029 %loss/total_loss‚40.25462420885764 % _timestamp‚1.7884349036368775e+09 2loss/SRqqt1/assignment_loss‚1.1223598458412765 1loss/SRqqt3/assignment_loss‚1.261137455701828 1loss/FBt1/assignment_loss‚0.06679769333147038 0loss/SRqqt3/detection_loss‚2.399172274505391 /loss/FBt2/detection_loss‚2.9915050268173218 /loss/FBt3/detection_loss‚2.7587380334734917 /loss/FRt2/assignment_loss‚2.639291853320842 .loss/FRt2/detection_loss‚2.289997266263378 1loss/SRqqt1/detection_loss‚0.8519326495411603 1loss/SRqqt4/detection_loss‚1.7424710124146705 /loss/FBt4/detection_loss‚2.4181307926774025 /loss/FRt1/detection_loss‚1.6086621356507143 /loss/FBt1/detection_loss‚0.6801728702048422 _runtime‚6588.341536817 /loss/FBt4/assignment_loss‚0.529832347529009 .loss/FRt3/detection_loss‚2.863467140708651 /loss/FRt4/detection_loss‚2.1182758590349784 0loss/SRqqt2/detection_loss‚1.857568118572235 epoch‚7 /loss/FRt4/assignment_loss‚3.043346960957234 2loss/SRqqt4/assignment_loss‚1.4501388823869181 _step‚716 /loss/FRt3/assignment_loss‚3.046826761109488 2loss/SRqqt2/assignment_loss‚1.4509207177907228 3loss/FBt2/assignment_loss‚6.377647649102869e-06 trainer/global_step‚8699 0loss/FBt3/assignment_loss‚0.2840731289811629¯®ÿˆ_É!Ù /loss/FBt4/assignment_loss‚0.529832347529009 .loss/FRt3/detection_loss‚2.863467140708651 /loss/FRt4/detection_loss‚2.1182758590349784 0loss/SRqqt2/detection_loss‚1.857568118572235 epoch‚7 /loss/FRt4/assignment_loss‚3.043346960957234 2loss/SRqqt4/assignment_loss‚1.4501388823869181 _step‚716 /loss/FRt3/assignment_loss‚3.046826761109488 2loss/SRqqt2/assignment_loss‚1.4509207177907228 3loss/FBt2/assignment_loss‚6.377647649102869e-06 trainer/global_step‚8699 0loss/FBt3/assignment_loss‚0.2840731289811629 0loss/FRt1/assignment_loss‚0.7797990043958029 %loss/total_loss‚40.25462420885764 % _timestamp‚1.7884349036368775e+09 2loss/SRqqt1/assignment_loss‚1.1223598458412765 1loss/SRqqt3/assignment_loss‚1.261137455701828 1loss/FBt1/assignment_loss‚0.06679769333147038 0loss/SRqqt3/detection_loss‚2.399172274505391 /loss/FBt2/detection_loss‚2.9915050268173218 /loss/FBt3/detection_loss‚2.7587380334734917 /loss/FRt2/assignment_loss‚2.639291853320842 .loss/FRt2/detection_loss‚2.289997266263378 1loss/SRqqt1/detection_loss‚0.8519326495411603 1loss/SRqqt4/detection_loss‚1.7424710124146705 /loss/FBt4/detection_loss‚2.4181307926774025 /loss/FRt1/detection_loss‚1.6086621356507143 /loss/FBt1/detection_loss‚0.6801728702048422 _runtime‚6588.341536817̵‹ ö“Ê!‚(:ˆ ã³åÔí·‡Ž gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö@|$Ë!‚(:™ ã³åÔ÷¿¿ disk.dm-0.out‚ 39570.8515625" disk.nvme0c0n1.in‚ 921.12890625$ memory_percent‚94.19240951538086) proc.memory.percent‚17.54765510559082 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 262.26953125 disk.rbd0.out‚0" proc.memory.rssMB‚ 2875.0078125 disk.dm-0.in‚ 921.12890625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 921.12890625 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0& proc.memory.availableMB‚ 951.515625 disk.dm-1.out‚ 262.26953125 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 network.recv‚ 2.23381e+06! disk.rbd2.out‚181.23583984375 disk.rbd3.out‚0% disk./.usageGB‚1100.8140296936035 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 39570.8515625 network.sent‚ 2.0424225e+07) disk./.usagePercent‚73.87655317722962" disk.nvme0n1.out‚ 39570.8515625 disk.rbd1.out‚ 3.373046875 disk.rbd0.in‚0 disk.sda3.out‚ 262.26953125 cpu‚25.362160957304397d+ç[’Ì!‚(:‡ ò³åÔ§¬÷ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚93.901' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`¦p#Í!‚(:˜ ò³åÔ뢉 disk.rbd3.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 262.640625# disk.nvme0n1.out‚39669.91796875% disk./.usageGB‚1100.8140296936035 disk.rbd1.in‚0" proc.memory.rssMB‚ 2875.0078125 cpu‚25.36145929245001% disk.nvme0c0n1.out‚39669.91796875 disk.rbd2.in‚0 disk.dm-0.in‚ 921.12890625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 3.373046875) proc.memory.percent‚17.54765510559082 network.sent‚ 2.0530383e+07 disk.sda3.in‚0( proc.memory.availableMB‚ 951.46484375 disk.drbd1777.out‚0 disk.sda.out‚ 262.640625) disk./.usagePercent‚73.87655317722962 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 921.12890625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.rbd2.out‚181.31396484375 disk.sda3.out‚ 262.640625 disk.sda.in‚0$ memory_percent‚94.19271945953369 network.recv‚ 2.238026e+06 disk.drbd1777.in‚0 disk.dm-0.out‚39669.91796875 disk.nvme0n1.in‚ 921.12890625 disk.sda2.in‚0 disk.sda2.out‚0ŸÁ2Î!- _wandbruntime‚6623  _runtime‚6623ÓœÝ@ÇÏ!Á & lr-AdamW/pg1‚4.875485051152286e-05 & lr-AdamW/pg2‚4.875485051152286e-05 trainer/global_step‚8749 % _timestamp‚1.7884349383981547e+09 _runtime‚6623.102898658 _step‚717¬šØœÌÐ!Æ & lr-AdamW/pg1‚4.875485051152286e-05 & lr-AdamW/pg2‚4.875485051152286e-05 trainer/global_step‚8749 % _timestamp‚1.7884349383981547e+09 _runtime‚6623.102898658 _step‚717ÍŽñ­â2Ñ!- _wandbruntime‚6623  _runtime‚6623iûÐ!ç'Ò!áO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292720337 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚718 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165287255 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚516681431997440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41883902 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚509348045106688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116228303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41884337 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚503297777527946 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0¨®t‹\!Ó!æO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚516682106362880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚556167808346624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚556167784406528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚8749 % _timestamp‚1.7884349383990142e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7307058243078 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165307293 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74343966 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7692532642816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚509348045106688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7306403805322 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127413044 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116227866 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚563860340989440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7333404893696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292720337 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚718 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165287255 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚516681431997440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41883902 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚509348045106688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116228303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41884337 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚503297777527946 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127427060 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292752833 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165305709 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116240572 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚509404398145536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116241027 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚509404398145536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74352132 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚503354169570794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚496046294892544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7693226234368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7334224321536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127427058 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292752833 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚563922851026432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74352134 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788434939055159e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7693391942144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚503353498093470 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7334880697344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7334224321536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚556229435144192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚8749 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165325773 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚496046277853184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚516738604464128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127427058 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚6623.760516948 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0M!f‹õ&Ö!ïM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7334224321536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127427058 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292752833 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚563922851026432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74352134 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788434939055159e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7693391942144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚503353498093470 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7334880697344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7334224321536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚556229435144192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚8749 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165325773 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚496046277853184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚516738604464128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127427058 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚6623.760516948 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127427060 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292752833 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165305709 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116240572 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚509404398145536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116241027 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚509404398145536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74352132 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚503354169570794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚496046294892544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7693226234368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844ÏfŸw2×!- _wandbruntime‚6623  _runtime‚6623ín t_Ø!Ù /loss/FBt4/detection_loss‚2.8166223069032035 2loss/SRqqt2/assignment_loss‚1.3085535776490966 /loss/FRt1/detection_loss‚1.7261565879452974 1loss/SRqqt4/detection_loss‚2.1203793474408084 /loss/FBt2/detection_loss‚2.3636510570844016 % _timestamp‚1.7884349390568924e+09 2loss/FBt3/assignment_loss‚0.001687707796615238 .loss/FBt3/detection_loss‚4.232212424278259 1loss/FBt2/assignment_loss‚0.17238680335867684 0loss/FBt4/assignment_loss‚0.3180437247501686 trainer/global_step‚8749 2loss/SRqqt1/assignment_loss‚1.0569303043386233 2loss/SRqqt3/assignment_loss‚1.8574176633321657 1loss/FBt1/assignment_loss‚0.14286682024839759 /loss/FRt4/detection_loss‚2.1611735481354954 0loss/SRqqt3/detection_loss‚2.218232226081011 /loss/FBt1/detection_loss‚0.5842415173137391 %loss/total_loss‚43.57572353289544 /loss/FRt1/assignment_loss‚1.450275334219138 0loss/FRt3/assignment_loss‚3.7640307661025756 0loss/FRt4/assignment_loss‚3.2517359372642307 2loss/SRqqt4/assignment_loss‚1.5588383782048558 0loss/SRqqt2/detection_loss‚1.590774198124806 0loss/FRt2/assignment_loss‚2.5140476953983306 /loss/FRt2/detection_loss‚2.1889780336618423 epoch‚7 _runtime‚6623.761503958 _step‚720 /loss/FRt3/detection_loss‚2.9032247143407024 1loss/SRqqt1/detection_loss‚1.2732628589229924™µÝ`dÙ!Þ 2loss/FBt3/assignment_loss‚0.001687707796615238 .loss/FBt3/detection_loss‚4.232212424278259 1loss/FBt2/assignment_loss‚0.17238680335867684 0loss/FBt4/assignment_loss‚0.3180437247501686 trainer/global_step‚8749 2loss/SRqqt1/assignment_loss‚1.0569303043386233 2loss/SRqqt3/assignment_loss‚1.8574176633321657 1loss/FBt1/assignment_loss‚0.14286682024839759 /loss/FRt4/detection_loss‚2.1611735481354954 0loss/SRqqt3/detection_loss‚2.218232226081011 /loss/FBt1/detection_loss‚0.5842415173137391 %loss/total_loss‚43.57572353289544 /loss/FRt1/assignment_loss‚1.450275334219138 0loss/FRt3/assignment_loss‚3.7640307661025756 0loss/FRt4/assignment_loss‚3.2517359372642307 2loss/SRqqt4/assignment_loss‚1.5588383782048558 0loss/SRqqt2/detection_loss‚1.590774198124806 0loss/FRt2/assignment_loss‚2.5140476953983306 /loss/FRt2/detection_loss‚2.1889780336618423 epoch‚7 _runtime‚6623.761503958 _step‚720 /loss/FRt3/detection_loss‚2.9032247143407024 1loss/SRqqt1/detection_loss‚1.2732628589229924 /loss/FBt4/detection_loss‚2.8166223069032035 2loss/SRqqt2/assignment_loss‚1.3085535776490966 /loss/FRt1/detection_loss‚1.7261565879452974 1loss/SRqqt4/detection_loss‚2.1203793474408084 /loss/FBt2/detection_loss‚2.3636510570844016 % _timestamp‚1.7884349390568924e+09Ð,…·žÚ!‚(:“ ´åÔëå÷ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.691' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.138199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_¬Ý¬Û!‚(:Œ ´åÔÐÉç disk.rbd0.in‚0 network.recv‚ 2.241965e+06 disk.rbd3.in‚0 disk.dm-1.out‚263.3125 disk.rbd1.in‚0$ memory_percent‚94.19293403625488 network.sent‚ 2.0535778e+07 disk.dm-0.out‚ 39769.859375 disk.rbd1.out‚ 3.373046875 disk.sda2.in‚0! disk.rbd2.out‚182.84130859375 disk.dm-0.in‚ 921.12890625 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚263.3125 disk.sda.out‚263.3125# disk.nvme0c0n1.out‚ 39769.859375 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 921.12890625" disk.nvme0c0n1.in‚ 921.12890625) disk./.usagePercent‚73.87662562736182' proc.memory.availableMB‚ 951.4296875) proc.memory.percent‚17.54765510559082% disk./.usageGB‚1100.8151092529297 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0! disk.nvme0n1.out‚ 39769.859375 disk.rbd2.in‚0" proc.memory.rssMB‚ 2875.0078125 cpu‚25.360713944270767 proc.cpu.threads‚17Co“Ü!‚(:ˆ ´åÔå¨õ gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.463' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü•,Ý!‚(:¡ ´åÔµÚŠ disk.rbd2.in‚0 disk.dm-0.in‚ 921.12890625 disk.sda1.in‚0 disk.sda2.in‚0 network.recv‚ 2.245672e+06 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 3.373046875 cpu‚25.360058527013102 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 921.12890625 proc.cpu.threads‚17% disk./.usageGB‚1100.8153686523438 disk.sda1.out‚0) disk./.usagePercent‚73.87664303587415 disk.sda.out‚ 263.6796875 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 921.12890625 disk.sda2.out‚0! disk.rbd2.out‚182.96630859375 disk.dm-1.out‚ 263.6796875% disk.nvme0n1.out‚39770.2314453125 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚94.19310092926025( proc.memory.availableMB‚ 951.40234375 network.sent‚ 2.064124e+07 disk.sda3.out‚ 263.6796875 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚39770.2314453125" proc.memory.rssMB‚ 2875.0078125) proc.memory.percent‚17.54765510559082 disk.rbd3.out‚0" disk.dm-0.out‚39770.2314453125 disk.sda.in‚0´b62Þ!- _wandbruntime‚6657  _runtime‚6657Φ¼Çß!Á & lr-AdamW/pg2‚4.741728482303664e-05 trainer/global_step‚8799 % _timestamp‚1.7884349732489548e+09 _runtime‚6657.953746623 _step‚721 & lr-AdamW/pg1‚4.741728482303664e-05 ¨]ºÌà!Æ _runtime‚6657.953746623 _step‚721 & lr-AdamW/pg1‚4.741728482303664e-05 & lr-AdamW/pg2‚4.741728482303664e-05 trainer/global_step‚8799 % _timestamp‚1.7884349732489548e+09Ñ—¬:ÿ2á!- _wandbruntime‚6657  _runtime‚6657Lêü‰â!áO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7374390699520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 trainer/global_step‚8799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚6657.954636789 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 % _timestamp‚1.7884349732496781e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166213307 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294327149 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚506084490872470 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚722 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚512165697049088 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116864467 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚519540744111104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7374390699520 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116864015 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128113842 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74752364 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inact‹ÌyŸ^ive_split_bytes.small_pool.allocated‚ 7735449470464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚512165679046144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74752366 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚566985794700288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559250345229824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7735290040832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚519540744111104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚512165697049088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166233345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128113842 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚498736597289984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7347893582486 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519540069745664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42111651 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚566985611330560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚506083819395354 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128113844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498736580250624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7347239144730 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7374390699520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 trainer/global_step‚8799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚6657.954636789 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 % _timestamp‚1.7884349732496781e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166213307 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294327149 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚506084490872470 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚722 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚512165697049088 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116864467 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚519540744111104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7374390699520 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116864015 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128113842 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74752364 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7735449470464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚512165679046144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74752366 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚566985794700288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559250345229824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7735290040832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚519540744111104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚512165697049088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166233345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128113842 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚498736597289984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7347893582486 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519540069745664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42111651 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚566985611330560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚506083819395354 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128113844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498736580250624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2Ò»X±Ì2ä!- _wandbruntime‚6658  _runtime‚6658©=žÁñ&å!ëM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74760532 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7348055579694 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚512222032084992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128127860 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7348710017658 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7375210127360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42116656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294359619 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519597916591104 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128127858 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116876730 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7736143023104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294379685 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166231761 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884349739040713e+09 _runtime‚6658.609228297 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294379685 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128127858 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519597916591104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚512222050087936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498791484381184 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128127860 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8799 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166231761 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7375866503168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74760534 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚559311995967488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚506140211438202 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7375210127360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7736310828032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498791501420544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚506139539960878 _step‚723 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚506140211438202 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7375210127360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7736310828032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498791501420544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚506139539960878 _step‚723 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7348710017658 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7375210127360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42116656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294359619 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519597916591104 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128127858 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116876730 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7736143023104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294379685 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166231761 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884349739040713e+09 _runtime‚6658.609228297 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294379685 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128127858 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519597916591104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚512222050087936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498791484381184 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128127860 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8799 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166231761 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7375866503168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74760534 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚559311995967488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414ÓÏ/(†2ç!- _wandbruntime‚6658  _runtime‚6658–þuÙ]è!× _runtime‚6658.610394243 2loss/SRqqt1/assignment_loss‚0.9429313708397169 /loss/FBt1/detection_loss‚0.5901698225503227 epoch‚7 2loss/SRqqt2/assignment_loss‚1.2585717713667286 1loss/FBt4/assignment_loss‚0.28453077633457724 /loss/FBt2/detection_loss‚2.1835723718007407 .loss/FBt4/detection_loss‚2.909835807979107 _step‚724 0loss/FRt3/assignment_loss‚3.5612715338215684 % _timestamp‚1.7884349739057748e+09 1loss/FBt1/assignment_loss‚0.10037084964445016 trainer/global_step‚8799 1loss/SRqqt3/detection_loss‚2.6757977467316847 1loss/SRqqt4/detection_loss‚1.8017637970772657 /loss/FRt2/detection_loss‚2.3239107356881195 1loss/FBt3/assignment_loss‚0.09616867049110377 .loss/FRt4/detection_loss‚2.341065103560686 %loss/total_loss‚41.92510397422123 /loss/FRt2/assignment_loss‚2.929720897719545 /loss/FRt4/assignment_loss‚2.576947782933712 /loss/FRt1/detection_loss‚1.7612160655887836 .loss/FRt3/detection_loss‚2.736476061922131 1loss/SRqqt1/detection_loss‚1.0621218676528623 1loss/SRqqt2/detection_loss‚1.7897485752900442 0loss/FRt1/assignment_loss‚1.7858007524875885 2loss/SRqqt3/assignment_loss‚1.9972078032218492 1loss/SRqqt4/assignment_loss‚1.270007567583922 1loss/FBt2/assignment_loss‚0.18581894966579662 /loss/FBt3/detection_loss‚2.7600772922689263(±”bé!Ü 1loss/SRqqt3/detection_loss‚2.6757977467316847 1loss/SRqqt4/detection_loss‚1.8017637970772657 /loss/FRt2/detection_loss‚2.3239107356881195 1loss/FBt3/assignment_loss‚0.09616867049110377 .loss/FRt4/detection_loss‚2.341065103560686 %loss/total_loss‚41.92510397422123 /loss/FRt2/assignment_loss‚2.929720897719545 /loss/FRt4/assignment_loss‚2.576947782933712 /loss/FRt1/detection_loss‚1.7612160655887836 .loss/FRt3/detection_loss‚2.736476061922131 1loss/SRqqt1/detection_loss‚1.0621218676528623 1loss/SRqqt2/detection_loss‚1.7897485752900442 0loss/FRt1/assignment_loss‚1.7858007524875885 2loss/SRqqt3/assignment_loss‚1.9972078032218492 1loss/SRqqt4/assignment_loss‚1.270007567583922 1loss/FBt2/assignment_loss‚0.18581894966579662 /loss/FBt3/detection_loss‚2.7600772922689263 _runtime‚6658.610394243 2loss/SRqqt1/assignment_loss‚0.9429313708397169 /loss/FBt1/detection_loss‚0.5901698225503227 epoch‚7 2loss/SRqqt2/assignment_loss‚1.2585717713667286 1loss/FBt4/assignment_loss‚0.28453077633457724 /loss/FBt2/detection_loss‚2.1835723718007407 .loss/FBt4/detection_loss‚2.909835807979107 _step‚724 0loss/FRt3/assignment_loss‚3.5612715338215684 % _timestamp‚1.7884349739057748e+09 1loss/FBt1/assignment_loss‚0.10037084964445016 trainer/global_step‚8799ÔÑž¾ê!‚(:’ Ÿ´åÔ¡„㎠gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.566000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00ý§1ë!‚(:¦ Ÿ´åÔ”†± disk.rbd3.in‚0 disk.sda.out‚ 264.24609375 disk.sda1.out‚0$ disk./.usageGB‚1100.815803527832" disk.nvme0c0n1.in‚ 921.12890625 disk.sda1.in‚0) proc.memory.percent‚17.54765510559082 network.recv‚ 2.248126e+06 disk.dm-1.out‚ 264.24609375& disk.nvme0n1.out‚39869.74560546875 disk.drbd1777.in‚0' proc.memory.availableMB‚ 951.4296875 network.sent‚ 2.0645767e+07 disk.dm-1.in‚0( disk.nvme0c0n1.out‚39869.74560546875! disk.rbd2.out‚182.98193359375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 921.12890625 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 921.12890625 cpu‚25.359323151111624 disk.rbd1.out‚ 3.373046875 disk.sda3.out‚ 264.24609375 disk.sda.in‚0" proc.memory.rssMB‚ 2875.0078125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0$ memory_percent‚94.19293403625488 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0# disk.dm-0.out‚39869.74560546875) disk./.usagePercent‚73.87667222073306i€æ›ì!‚(: ®´åÔüÞŒŽ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.29' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.458000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ÊÇ)í!‚(:ž ®´åÔìÞˆ’ network.sent‚ 2.0751308e+07 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 921.12890625 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 264.6484375& disk.nvme0n1.out‚39968.96533203125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 183.068359375 disk.dm-1.out‚ 264.6484375$ disk./.usageGB‚1100.815803527832# disk.dm-0.out‚39968.96533203125 disk.sda.in‚0( disk.nvme0c0n1.out‚39968.96533203125 disk.drbd1777.in‚0) proc.memory.percent‚17.54765510559082 disk.sda1.out‚0 disk.nvme0n1.in‚ 921.12890625 disk.rbd1.in‚0 disk.rbd1.out‚ 3.373046875 disk.sda2.out‚0" proc.memory.rssMB‚ 2875.0078125 proc.cpu.threads‚17 network.recv‚ 2.251734e+06 disk.rbd0.out‚0% proc.memory.availableMB‚ 951.34375) disk./.usagePercent‚73.87667222073306 disk.sda3.out‚ 264.6484375 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 921.12890625 disk.dm-1.in‚0# memory_percent‚94.1934585571289 cpu‚25.358654897394764 disk.rbd0.in‚0:®¢ “î!‚(:ˆ ½´åÔ²»—Ž gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.719' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9438 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÙ,þ,ï!‚(:¡ ½´åÔù±þ) disk./.usagePercent‚73.87668322905702 disk.sda1.out‚0 disk.dm-1.in‚0 cpu‚25.35793664458606 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0# disk.dm-0.out‚40069.17138671875 disk.sda.in‚0 disk.rbd1.out‚ 3.373046875 disk.sda1.in‚0& disk.nvme0n1.out‚40069.17138671875# memory_percent‚94.1934585571289" proc.memory.rssMB‚ 2875.0078125 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 921.12890625 disk.sda2.in‚0 disk.rbd0.in‚0% proc.memory.availableMB‚ 951.34375 disk.rbd2.out‚ 183.083984375 disk.nvme0n1.in‚ 921.12890625 proc.cpu.threads‚17( disk.nvme0c0n1.out‚40069.17138671875 disk.dm-0.in‚ 921.12890625 disk.dm-1.out‚ 265.23828125 network.sent‚ 2.0757671e+07 network.recv‚ 2.256224e+06% disk./.usageGB‚1100.8159675598145 disk.sda3.out‚ 265.23828125 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 265.23828125) proc.memory.percent‚17.54765510559082x2ã:2ð!- _wandbruntime‚6693  _runtime‚6693ñhÄñ!¾ trainer/global_step‚8849 # _timestamp‚1.78843500836732e+09 _runtime‚ 6693.07206864 _step‚725 & lr-AdamW/pg1‚4.609345137008857e-05 & lr-AdamW/pg2‚4.609345137008857e-05ÐÒ`Éò!à # _timestamp‚1.78843500836732e+09 _runtime‚ 6693.07206864 _step‚725 & lr-AdamW/pg1‚4.609345137008857e-05 & lr-AdamW/pg2‚4.609345137008857e-05 trainer/global_step‚8849Õ Ezp2ó!- _wandbruntime‚6693  _runtime‚6693µR˜ªç'ô!áO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚508870532739878 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42340142 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚562332882113024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7778528253952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7778370921472 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117501371 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295954001 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128814644 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167159397 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128814642 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7416032867840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128814642 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514983348991488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514983348991488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884350083681176e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚570111229094400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562332858172928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚522399381859328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75160764 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7388074484138 _runtime‚6693.073035842 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295974041 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501481786778624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117500906 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚522399381859328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚501481803817984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75160766 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚514983330988544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚508869861262762 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7388728921894 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522398707493888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7416032867840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7415376505344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚726 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128814644 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚8849 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514983330988544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42340605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7415376505344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42340142 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚562332882113024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7778528253952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7778370921472 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117501371 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295954001 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128814644 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167159397 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128814642 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7416032867840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128814642 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514983348991488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514983348991488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884350083681176e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚570111229094400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562332858172928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚522399381859328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75160764 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7388074484138 _runtime‚6693.073035842 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295974041 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501481786778624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117500906 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚522399381859328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚501481803817984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75160766 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚514983330988544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚508869861262762 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7388728921894 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522398707493888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7416032867840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7415376505344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚726ÖÜѯ2ö!- _wandbruntime‚6693  _runtime‚6693Ëß‹Ä÷!êM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296006537 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522456554339328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128828660 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167177877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚727 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167157813 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviÅ~,!ceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128828658 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚562394532850688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7416195933184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75168934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚515039702030336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75168932 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42344677 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7779243826688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚508926253305610 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚501536690909184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7779401145856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7416852308992 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128828658 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚515039702030336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚570173933996544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296006537 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167157813 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7389545357066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167177877 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7416852308992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7416195933184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7388890919102 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128828660 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚515039684027392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚508925581828286 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚501536707948544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚515039684027392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117513609 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚562394508910592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295986471 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117513609 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚562394508910592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295986471 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522456554339328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128828660 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167177877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚727 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167157813 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128828658 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚562394532850688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7416195933184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75168934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚515039702030336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75168932 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42344677 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7779243826688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚508926253305610 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚501536690909184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7779401145856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7416852308992 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128828658 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚515039702030336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚570173933996544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296006537 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167157813 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7389545357066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167177877 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7416852308992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7416195933184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7388890919102 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128828660 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚515039684027392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚508925581828286 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚501536707948544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414×up\2ù!- _wandbruntime‚6693  _runtime‚6693cÜÎOZú!Ô 1loss/FBt3/assignment_loss‚0.08125095602864509 .loss/FRt2/detection_loss‚2.385640211742033 1loss/SRqqt4/detection_loss‚1.9126636627174558 0loss/FRt2/assignment_loss‚2.5227957462722603 %loss/total_loss‚41.40647322181371 trainer/global_step‚8849 2loss/SRqqt1/assignment_loss‚0.9902381113183404 1loss/SRqqt4/assignment_loss‚1.403665130514474 .loss/FBt3/detection_loss‚2.618184475337758 1loss/SRqqt3/detection_loss‚2.0722087258245887 0loss/FRt3/assignment_loss‚3.6251878142356873 0loss/FRt4/assignment_loss‚2.9667926791467165 0loss/FBt4/assignment_loss‚0.4868624080166531 .loss/FRt1/detection_loss‚1.551567022565682 .loss/FRt3/detection_loss‚2.810815857302758 .loss/FRt4/detection_loss‚2.510517880320549 1loss/SRqqt2/detection_loss‚1.8614208709637998 1loss/SRqqt3/assignment_loss‚1.833397874018041 2loss/SRqqt2/assignment_loss‚1.1541131589624842 3loss/FBt2/assignment_loss‚0.0024026803413497483 1loss/SRqqt1/detection_loss‚0.8569425809832792 /loss/FBt1/detection_loss‚0.6608900510583002 /loss/FBt2/detection_loss‚2.9856480104582652 epoch‚7 0loss/FRt1/assignment_loss‚1.1857997921277892 0loss/FBt1/assignment_loss‚0.1388702308422235 _runtime‚6693.744250981 _step‚728 .loss/FBt4/detection_loss‚2.788597290714582 $ _timestamp‚1.788435009039616e+09+é4•_û!Ù 0loss/FRt1/assignment_loss‚1.1857997921277892 0loss/FBt1/assignment_loss‚0.1388702308422235 _runtime‚6693.744250981 _step‚728 .loss/FBt4/detection_loss‚2.788597290714582 $ _timestamp‚1.788435009039616e+09 1loss/FBt3/assignment_loss‚0.08125095602864509 .loss/FRt2/detection_loss‚2.385640211742033 1loss/SRqqt4/detection_loss‚1.9126636627174558 0loss/FRt2/assignment_loss‚2.5227957462722603 %loss/total_loss‚41.40647322181371 trainer/global_step‚8849 2loss/SRqqt1/assignment_loss‚0.9902381113183404 1loss/SRqqt4/assignment_loss‚1.403665130514474 .loss/FBt3/detection_loss‚2.618184475337758 1loss/SRqqt3/detection_loss‚2.0722087258245887 0loss/FRt3/assignment_loss‚3.6251878142356873 0loss/FRt4/assignment_loss‚2.9667926791467165 0loss/FBt4/assignment_loss‚0.4868624080166531 .loss/FRt1/detection_loss‚1.551567022565682 .loss/FRt3/detection_loss‚2.810815857302758 .loss/FRt4/detection_loss‚2.510517880320549 1loss/SRqqt2/detection_loss‚1.8614208709637998 1loss/SRqqt3/assignment_loss‚1.833397874018041 2loss/SRqqt2/assignment_loss‚1.1541131589624842 3loss/FBt2/assignment_loss‚0.0024026803413497483 1loss/SRqqt1/detection_loss‚0.8569425809832792 /loss/FBt1/detection_loss‚0.6608900510583002 /loss/FBt2/detection_loss‚2.9856480104582652 epoch‚7Øìø&”žü!‚(:“ Ì´åÔαÛ gpu.0.gpu‚33.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.658' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.531599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤”~áý!‚(:” Ì´åÔ¡“˜‘ cpu‚25.35724896904346 disk.rbd1.out‚ 3.373046875% disk./.usageGB‚1100.8160247802734 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 184.720703125 proc.cpu.threads‚17% disk.nvme0n1.out‚40168.2666015625 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2875.0078125 network.recv‚ 2.258888e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 265.8671875' disk.nvme0c0n1.out‚40168.2666015625 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.nvme0c0n1.in‚ 921.171875 disk.sda2.in‚0 disk.drbd1777.out‚0$ memory_percent‚94.19326782226562# proc.memory.availableMB‚951.375) proc.memory.percent‚17.54765510559082" disk.dm-0.out‚40168.2666015625 disk.rbd1.in‚0 network.sent‚ 2.0762096e+07) disk./.usagePercent‚73.87668706917005 disk.sda3.in‚0 disk.dm-0.in‚ 921.171875 disk.sda.out‚ 265.8671875 disk.nvme0n1.in‚ 921.171875 disk.sda2.out‚0 disk.sda3.out‚ 265.8671875ʤa ‘þ!‚(:† Û´åÔ“ªÚ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚93.295' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.659 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:u#ÿ!‚(:˜ Û´åÔ½…‘ disk.sda3.in‚0 disk.dm-1.out‚ 266.390625& disk.nvme0n1.out‚40169.95751953125 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.rbd2.out‚ 184.736328125% disk./.usageGB‚1100.8158721923828 disk.sda1.in‚0( disk.nvme0c0n1.out‚40169.95751953125 disk.rbd0.in‚0$ memory_percent‚94.19357776641846( proc.memory.availableMB‚ 951.32421875 network.recv‚ 2.263795e+06 disk.sda2.out‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 921.171875 disk.nvme0c0n1.in‚ 921.171875 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.373046875 disk.sda2.in‚0) proc.memory.percent‚17.54765510559082 disk.dm-0.in‚ 921.171875" proc.memory.rssMB‚ 2875.0078125 proc.cpu.threads‚17) disk./.usagePercent‚73.87667682886867 disk.sda3.out‚ 266.390625# disk.dm-0.out‚40169.95751953125 disk.sda.out‚ 266.390625 cpu‚25.3565669241635 network.sent‚ 2.0869712e+07mžã2€"- _wandbruntime‚6727  _runtime‚6727¬ZÃtÉ"à _step‚729 ' lr-AdamW/pg1‚4.4783626037586085e-05 ' lr-AdamW/pg2‚4.4783626037586085e-05 trainer/global_step‚8899 % _timestamp‚1.7884350432843068e+09 _runtime‚6727.989119699¡Ë΂"È ' lr-AdamW/pg1‚4.4783626037586085e-05 ' lr-AdamW/pg2‚4.4783626037586085e-05 trainer/global_step‚8899 % _timestamp‚1.7884350432843068e+09 _runtime‚6727.989119699 _step‚729Ùp©2ƒ"- _wandbruntime‚6727  _runtime‚6727µÁ &„"àO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚517800982930944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42567578 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚504227010345984 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚504226993306624 _step‚730 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚525257345242112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75569164 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7429564261302 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168085449 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129515444 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚565415418996224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7456362311168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168085449 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚511656574607286 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚511655903130170 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7821662991360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7457018673664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7457018673664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚6727.990088715 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75569166 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚517800982930944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚517801000933888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7821493075968qþ£ë'…"åO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7456362311168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168085449 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚511656574607286 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚511655903130170 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7821662991360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7457018673664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7457018673664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚6727.990088715 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75569166 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚517800982930944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚517801000933888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7821493075968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚517800982930944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42567578 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚504227010345984 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚504226993306624 _step‚730 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚525257345242112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75569164 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7429564261302 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168085449 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129515444 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚565415418996224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2Ú\2†"- _wandbruntime‚6728  _runtime‚6728R]gñ&‡"ëM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7457181739008 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129529458 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42572024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚525315192087552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42571587 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚504281914476544 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168103929 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7457181739008 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168083865 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297633389 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚517857353972736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297633389 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚511711623695694 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚565477069733888 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚6728.659585118 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118148919 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚8899 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚517857335969792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚731 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚573299415137792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚525314517708800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168103929 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7429726258510 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7457838114816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7822522468864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚511712295173018 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7430380696474 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129529460 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚565477045793792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7822369344000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75577332 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884350439544635e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168083865 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7457838114816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297613323 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7457181739008 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168083865 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297633389 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚517857353972736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297633389 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚511711623695694 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚565477069733888 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚6728.659585118 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118148919 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚8899 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚517857335969792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚731 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚573299415137792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚525314517708800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168103929 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7429726258510 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7457838114816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7822522468864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚511712295173018 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7430380696474 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129529460 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚565477045793792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7822369344000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75577332 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884350439544635e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168083865 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7457838114816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297613323 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168991463 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚576362446891008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75977564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130216244 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚520618634873344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7498004479488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130216242 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169011501 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299227745 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚568497955879424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506972216873984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42794746 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7497348116992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚576362642843648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7469745162954 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚528115982990336 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚568497931939328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚520618652876288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7498004479488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118772312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75977566 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7864514951680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130216242 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7497348116992 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130216244 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7864686964224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚520618634873344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚506972199834624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42794292 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _step‚734 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884350784901643e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7470399600710 trainer/global_step‚8949 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚ 6763.19508597(K¬þë'•"åO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚528115982990336 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚568497931939328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚520618652876288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7498004479488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118772312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75977566 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7864514951680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130216242 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7497348116992 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130216244 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7864686964224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚520618634873344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚506972199834624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42794292 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _step‚734 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884350784901643e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7470399600710 trainer/global_step‚8949 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚ 6763.19508597 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚528115982990336 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168991463 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚528116657355776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118771856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169011501 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚514442616474694 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168991463 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚576362446891008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75977564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130216244 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚520618634873344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7498004479488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130216242 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169011501 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299227745 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚568497955879424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506972216873984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42794746 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7497348116992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚576362642843648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7469745162954Þn±Ñ2–"- _wandbruntime‚6763  _runtime‚6763&k×]F—"ëM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130230260 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130230260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚528173829835776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42798862 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚520674987912192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169009917 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚6763.855910341 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130230258 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚507027121004544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚568559582676992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚528173155457024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚568559606617088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884350791507175e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚576425147844608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚520675005915136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7470561597918 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚520675005915136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7498823920640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299240175 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118785043 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7865379714048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169029981 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚520674987912192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75985732 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7865541227520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299260241 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576424962391040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130230258 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _step‚735 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚514497665563102 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_tra×ÊÙ‹«in_batch_end/num_alloc_retries‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42799309 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7498167544832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299260241 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118784594 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299240175 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7498823920640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚514498337040426 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7471216035882 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430P½S«ö&˜"ðM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚514497665563102 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42799309 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7498167544832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299260241 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118784594 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299240175 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7498823920640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚514498337040426 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7471216035882 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130230260 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130230260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚528173829835776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42798862 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚520674987912192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169009917 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚6763.855910341 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130230258 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚507027121004544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚568559582676992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚528173155457024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚568559606617088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884350791507175e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚576425147844608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚520675005915136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7470561597918 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚520675005915136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7498823920640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299240175 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118785043 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7865379714048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169029981 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚520674987912192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75985732 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7865541227520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299260241 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576424962391040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130230258 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _step‚735ß‹‹j2™"- _wandbruntime‚6763  _runtime‚6763ÎÉSš"Í .loss/FRt3/assignment_loss‚3.42850627047675 .loss/FRt2/detection_loss‚2.345204201837381 $ _timestamp‚1.788435079152261e+09 _runtime‚ 6763.85721027 .loss/FRt1/detection_loss‚1.333068900232045 1loss/SRqqt2/detection_loss‚1.8657210594415665 /loss/FBt1/detection_loss‚0.5336308764575127 .loss/FBt4/detection_loss‚2.355463649545397 1loss/SRqqt1/assignment_loss‚0.789881092224034 /loss/FRt3/detection_loss‚2.8164412191935946 0loss/FRt1/assignment_loss‚1.2986716661813125 /loss/FRt4/assignment_loss‚2.703536768004579 2loss/SRqqt2/assignment_loss‚1.3339486233890057 0loss/SRqqt4/detection_loss‚1.917819284378214 epoch‚7 0loss/FBt2/assignment_loss‚0.4514714009128511 .loss/FBt3/detection_loss‚2.191441059112549 _step‚736 trainer/global_step‚8949 .loss/FRt2/assignment_loss‚2.47076503559947 1loss/FBt1/assignment_loss‚0.08753475765791856 1loss/FBt4/assignment_loss‚0.02314781439158001 /loss/FBt2/detection_loss‚2.4968120455741882 2loss/SRqqt3/assignment_loss‚1.3288568675737171 0loss/FBt3/assignment_loss‚0.9237076069694012 .loss/FRt4/detection_loss‚2.118552311411444 1loss/SRqqt3/detection_loss‚2.3376648531240574 2loss/SRqqt4/assignment_loss‚1.3072313321556182 1loss/SRqqt1/detection_loss‚0.8114374802041222 %loss/total_loss‚39.27051617604831KÛE:X›"Ò 2loss/SRqqt3/assignment_loss‚1.3288568675737171 0loss/FBt3/assignment_loss‚0.9237076069694012 .loss/FRt4/detection_loss‚2.118552311411444 1loss/SRqqt3/detection_loss‚2.3376648531240574 2loss/SRqqt4/assignment_loss‚1.3072313321556182 1loss/SRqqt1/detection_loss‚0.8114374802041222 %loss/total_loss‚39.27051617604831 .loss/FRt3/assignment_loss‚3.42850627047675 .loss/FRt2/detection_loss‚2.345204201837381 $ _timestamp‚1.788435079152261e+09 _runtime‚ 6763.85721027 .loss/FRt1/detection_loss‚1.333068900232045 1loss/SRqqt2/detection_loss‚1.8657210594415665 /loss/FBt1/detection_loss‚0.5336308764575127 .loss/FBt4/detection_loss‚2.355463649545397 1loss/SRqqt1/assignment_loss‚0.789881092224034 /loss/FRt3/detection_loss‚2.8164412191935946 0loss/FRt1/assignment_loss‚1.2986716661813125 /loss/FRt4/assignment_loss‚2.703536768004579 2loss/SRqqt2/assignment_loss‚1.3339486233890057 0loss/SRqqt4/detection_loss‚1.917819284378214 epoch‚7 0loss/FBt2/assignment_loss‚0.4514714009128511 .loss/FBt3/detection_loss‚2.191441059112549 _step‚736 trainer/global_step‚8949 .loss/FRt2/assignment_loss‚2.47076503559947 1loss/FBt1/assignment_loss‚0.08753475765791856 1loss/FBt4/assignment_loss‚0.02314781439158001 /loss/FBt2/detection_loss‚2.4968120455741882àâ:™žœ"‚(:“ ˆµåÔ‰ÉÌ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚132.284' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.456799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äåDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530974620738560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚571580468822528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7538333922816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43021777 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130917044 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚579487921870848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7907608283648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76385966 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚523436304818688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130917042 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7511234940118 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300834557 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚571580492762624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517228658342102 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ÔâþBö§"æO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517228658342102 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300854597 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7538333922816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_sJ`Ë»ö#tart/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚530974620738560 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚579488101046272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530974620738560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚571580468822528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7538333922816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43021777 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130917044 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚579487921870848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7907608283648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76385966 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚523436304818688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130917042 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7511234940118 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300834557 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚571580492762624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038âOOù2¨"- _wandbruntime‚6798  _runtime‚6798s>ÿ‚ñ&©"ëM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130931060 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚523492639854592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚579550446508032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76394132 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169935969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚6798.943938868 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130931058 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119420898 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571642143500288 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43026333 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7908326947840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517283707430510 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169956033 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523492657857536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7539153350656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119420465 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7908482169856 trainer/global_step‚8999 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517284378907834 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509772310493184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300887093 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300867027 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169956033 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300867027 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7539153350656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚571642119560192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚531032467584000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76394134 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7539809726464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884351142385375e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169935969 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300887093 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952†àõäö&ª"ðM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚531031793205248 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300887093 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130931060 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚523492639854592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚579550446508032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76394132 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169935969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚6798.943938868 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130931058 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119420898 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571642143500288 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43026333 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7908326947840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517283707430510 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169956033 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523492657857536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7539153350656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119420465 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7908482169856 trainer/global_step‚8999 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517284378907834 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509772310493184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300887093 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300867027 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169956033 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300867027 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7539153350656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚571642119560192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚531032467584000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76394134 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7539809726464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884351142385375e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169935969 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚582613505967616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302481449 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚533833258486784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚520014700209510 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚520014028732394 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170863605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚526253938758144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302461409 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚526253938758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚512462629929984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚6833.151070656 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170843567 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7579319728640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76794366 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131617844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚533833258486784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7579976091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚742 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚512462612890624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚582613693531648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚574663029645824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚526253956761088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚533833258486784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚520014700209510 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚520014028732394 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170863605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚526253938758144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302461409 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚526253938758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚512462629929984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚6833.151070656 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170843567 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7579319728640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76794366 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131617844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚533833258486784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7579976091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚742 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚512462612890624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚582613693531648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚574663029645824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚526253956761088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚582613505967616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302481449æ> O(2¸"- _wandbruntime‚6833  _runtime‚6833ö?Óg%¹"ëM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131631858 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131631860 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7580795532288 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6833.807034584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚526310291796992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302513945 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76802534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7552886714698 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚9049 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚533891105332224 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7580795532288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43254577 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚582676024890880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7951529960960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚512517534060544 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302493879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43255005 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7580139156480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120057109 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚582676210344448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7951368447488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120057539 _step‚743 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302513945 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170862021 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚574724656443392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302493879 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚533890430953472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚526310291796992ÆéŠö&º"ðM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43255005 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7580139156480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120057109 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚582676210344448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7951368447488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120057539 _step‚743 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302513945 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170862021 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚574724656443392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302493879 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚533890430953472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚526310291796992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131631858 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131631860 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7580795532288 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6833.807034584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚526310291796992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302513945 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76802534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7552886714698 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚9049 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚533891105332224 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7580795532288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43254577 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚582676024890880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7951529960960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚512517534060544 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302493879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472çJ×Öb2»"- _wandbruntime‚6833  _runtime‚6833÷䲨[¼"Õ 1loss/FBt1/assignment_loss‚0.13775986748994337 1loss/FBt3/assignment_loss‚0.09295676156346287 0loss/SRqqt3/detection_loss‚2.505167874796637 /loss/FBt4/detection_loss‚3.1430174112319946 trainer/global_step‚9049 /loss/FRt3/assignment_loss‚2.986418980818528 _step‚744 2loss/SRqqt3/assignment_loss‚1.7738508591749544 2loss/SRqqt4/assignment_loss‚1.4111245335188023 .loss/FRt4/detection_loss‚2.057892025758823 1loss/SRqqt1/detection_loss‚0.8072618915484502 % _timestamp‚1.7884351491033587e+09 0loss/FBt2/assignment_loss‚0.1514135309585577 _runtime‚6833.808012776 0loss/FRt1/assignment_loss‚1.0745317747195562 0loss/FBt4/assignment_loss‚1.0956055129645392 /loss/FRt1/detection_loss‚1.5052918846408525 /loss/FRt3/detection_loss‚2.9185970035883098 %loss/total_loss‚41.17921512998228 0loss/FRt4/assignment_loss‚3.2700809439023337 2loss/SRqqt2/assignment_loss‚1.6054064546977835 1loss/SRqqt4/detection_loss‚1.9272257741107497 /loss/FBt2/detection_loss‚2.0100829071468778 .loss/FRt2/detection_loss‚2.336371249478796 0loss/FRt2/assignment_loss‚2.6393225374429123 .loss/FBt1/detection_loss‚0.541706512636149 2loss/SRqqt1/assignment_loss‚0.7489375006506005 1loss/SRqqt2/detection_loss‚1.7774228836808885 .loss/FBt3/detection_loss‚2.661768453461783 epoch‚7ŒÓÐÆ`½"Ú /loss/FRt3/assignment_loss‚2.986418980818528 _step‚744 2loss/SRqqt3/assignment_loss‚1.7738508591749544 2loss/SRqqt4/assignment_loss‚1.4111245335188023 .loss/FRt4/detection_loss‚2.057892025758823 1loss/SRqqt1/detection_loss‚0.8072618915484502 % _timestamp‚1.7884351491033587e+09 0loss/FBt2/assignment_loss‚0.1514135309585577 _runtime‚6833.808012776 0loss/FRt1/assignment_loss‚1.0745317747195562 0loss/FBt4/assignment_loss‚1.0956055129645392 /loss/FRt1/detection_loss‚1.5052918846408525 /loss/FRt3/detection_loss‚2.9185970035883098 %loss/total_loss‚41.17921512998228 0loss/FRt4/assignment_loss‚3.2700809439023337 2loss/SRqqt2/assignment_loss‚1.6054064546977835 1loss/SRqqt4/detection_loss‚1.9272257741107497 /loss/FBt2/detection_loss‚2.0100829071468778 .loss/FRt2/detection_loss‚2.336371249478796 0loss/FRt2/assignment_loss‚2.6393225374429123 .loss/FBt1/detection_loss‚0.541706512636149 2loss/SRqqt1/assignment_loss‚0.7489375006506005 1loss/SRqqt2/detection_loss‚1.7774228836808885 .loss/FBt3/detection_loss‚2.661768453461783 epoch‚7 1loss/FBt1/assignment_loss‚0.13775986748994337 1loss/FBt3/assignment_loss‚0.09295676156346287 0loss/SRqqt3/detection_loss‚2.505167874796637 /loss/FBt4/detection_loss‚3.1430174112319946 trainer/global_step‚9049è|Q?“¾"‚(:ˆ ÓµåÔð¼û gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.5238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wQ ¿"‚(:ˆ Óµåԙť‘ disk.sda.in‚0 disk.drbd1777.in‚0" proc.memory.rssMB‚ 2875.0078125) proc.memory.percent‚17.54765510559082 disk.sda1.out‚0 disk.nvme0n1.out‚ 40766.625 disk.sda3.in‚0 disk.sda3.out‚ 270.9296875 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 921.171875 disk.rbd3.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1100.8164291381836 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚ 40766.625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 cpu‚25.351134155946298) disk./.usagePercent‚73.87671420596867 disk.sda.out‚ 270.9296875 disk.rbd1.in‚0 disk.rbd1.out‚ 3.61669921875( proc.memory.availableMB‚ 950.87109375 network.sent‚ 2.1210925e+07 disk.sda2.out‚0 disk.rbd2.out‚186.6513671875 disk.dm-0.in‚ 921.171875$ memory_percent‚94.19634342193604 network.recv‚ 2.288755e+06 disk.dm-1.out‚ 270.9296875 disk.nvme0c0n1.in‚ 921.171875 disk.nvme0c0n1.out‚ 40766.625ÆkߧžÀ"‚(:“ âµåÔëüÿ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.545' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.908999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T+Z#Á"‚(:˜ âµåÔ«Ó±‘ disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 921.171875 disk.sda.in‚0) proc.memory.percent‚17.54765510559082 disk.rbd0.in‚0 network.sent‚ 2.1317835e+07) disk./.usagePercent‚73.87672239820976 disk.rbd1.out‚ 3.6796875 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 271.5546875$ memory_percent‚94.19488906860352 disk.nvme0n1.in‚ 921.171875 disk.rbd2.out‚186.7919921875 disk.drbd1777.in‚0 cpu‚25.35047697757636$ disk./.usageGB‚1100.816551208496 disk.rbd3.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚40865.84130859375# disk.dm-0.out‚40865.84130859375 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0& proc.memory.availableMB‚ 951.109375" proc.memory.rssMB‚ 2875.0078125 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 921.171875 disk.sda1.out‚0 disk.rbd0.out‚0 network.recv‚ 2.292635e+06( disk.nvme0c0n1.out‚40865.84130859375 disk.sda3.out‚ 271.5546875 disk.sda.out‚ 271.5546875&½@d2Â"- _wandbruntime‚6868  _runtime‚6868ž²ÉÃ"à ' lr-AdamW/pg1‚3.9689820245438464e-05 ' lr-AdamW/pg2‚3.9689820245438464e-05 trainer/global_step‚9099 % _timestamp‚1.7884351835225494e+09 _runtime‚6868.227298634 _step‚745ôiðqÎÄ"È ' lr-AdamW/pg1‚3.9689820245438464e-05 ' lr-AdamW/pg2‚3.9689820245438464e-05 trainer/global_step‚9099 % _timestamp‚1.7884351835225494e+09 _runtime‚6868.227298634 _step‚745éÝ|*~2Å"- _wandbruntime‚6868  _runtime‚68688ù¨ç'Æ"áO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚6868.228159185 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚529071608703488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7620961896960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7620305534464 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7620961896960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7620305534464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77202766 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43478232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚522800742076918 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171789657 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120680998 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120680531 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚536691896235008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚577745566529024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚522800070599802 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7592905618934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚577745542588928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7592251181178 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚529071590700544 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132318642 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7993589462528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚536691896235008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120680998 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120680531 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚536691896235008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚577745566529024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚522800070599802 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7592905618934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚577745542588928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7592251181178 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚529071590700544 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132318642 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7993589462528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚536691896235008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304120731 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7994430512128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171808137 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77210934 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7593722054106 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚515262723549184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120693719 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304140797 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171788073 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7994606705664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚747 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529127943739392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7621781338112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132332660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77210932 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 # _timestamp‚1.78843518418326e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7621781338112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚536749743080448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529127961742336 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚522856462642650 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120693244 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚536749743080448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚522855791165326 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚9099 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304140797 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43482312 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132332658 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304140797 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171788073 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7994606705664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚747 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529127943739392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7621781338112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132332660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77210932 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 # _timestamp‚1.78843518418326e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7621781338112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚536749743080448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529127961742336 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚522856462642650 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120693244 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚536749743080448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚522855791165326 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚9099 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304140797 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43482312 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132332658 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304120731 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7994430512128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171808137 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77210934 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7593722054106ë+ÿÿJ2Ë"- _wandbruntime‚6868  _runtime‚6868¸­Iw_Ì"Ù /loss/FRt2/assignment_loss‚2.863500552681776 2loss/FBt1/assignment_loss‚0.035677089275116994 1loss/SRqqt4/detection_loss‚1.7480605536928544 /loss/FRt4/detection_loss‚2.4180074368798454 epoch‚7 2loss/SRqqt3/assignment_loss‚1.3847418148760442 /loss/FBt4/detection_loss‚2.1657433774736194 &loss/total_loss‚39.872629496423045 % _timestamp‚1.7884351841850235e+09 1loss/FBt3/assignment_loss‚0.09113686410710216 1loss/SRqqt2/detection_loss‚1.8412137911827475 1loss/SRqqt3/detection_loss‚2.5187446695786937 2loss/SRqqt2/assignment_loss‚1.4843704826802153 1loss/SRqqt1/detection_loss‚0.8671847557331004 0loss/FRt1/assignment_loss‚1.0605759718379035 /loss/FRt4/assignment_loss‚2.513489326765371 1loss/FBt4/assignment_loss‚0.13546902168956068 /loss/FRt1/detection_loss‚1.5459212366430486 .loss/FRt2/detection_loss‚2.342548741744115 /loss/FRt3/detection_loss‚3.1349760385660024 1loss/SRqqt1/assignment_loss‚0.802983548832921 2loss/SRqqt4/assignment_loss‚1.3039035539214427 1loss/FBt2/assignment_loss‚0.27955203712086324 /loss/FBt1/detection_loss‚0.5091233993326568 /loss/FBt2/detection_loss‚2.4176450882639204 .loss/FBt3/detection_loss‚3.079160237312317 trainer/global_step‚9099 _runtime‚ 6868.88965894 0loss/FRt3/assignment_loss‚3.3288999062318068 _step‚748ÃôûdÍ"Þ /loss/FRt4/detection_loss‚2.4180074368798454 epoch‚7 2loss/SRqqt3/assignment_loss‚1.3847418148760442 /loss/FBt4/detection_loss‚2.1657433774736194 &loss/total_loss‚39.872629496423045 % _timestamp‚1.7884351841850235e+09 1loss/FBt3/assignment_loss‚0.09113686410710216 1loss/SRqqt2/detection_loss‚1.8412137911827475 1loss/SRqqt3/detection_loss‚2.5187446695786937 2loss/SRqqt2/assignment_loss‚1.4843704826802153 1loss/SRqqt1/detection_loss‚0.8671847557331004 0loss/FRt1/assignment_loss‚1.0605759718379035 /loss/FRt4/assignment_loss‚2.513489326765371 1loss/FBt4/assignment_loss‚0.13546902168956068 /loss/FRt1/detection_loss‚1.5459212366430486 .loss/FRt2/detection_loss‚2.342548741744115 /loss/FRt3/detection_loss‚3.1349760385660024 1loss/SRqqt1/assignment_loss‚0.802983548832921 2loss/SRqqt4/assignment_loss‚1.3039035539214427 1loss/FBt2/assignment_loss‚0.27955203712086324 /loss/FBt1/detection_loss‚0.5091233993326568 /loss/FBt2/detection_loss‚2.4176450882639204 .loss/FBt3/detection_loss‚3.079160237312317 trainer/global_step‚9099 _runtime‚ 6868.88965894 0loss/FRt3/assignment_loss‚3.3288999062318068 _step‚748 /loss/FRt2/assignment_loss‚2.863500552681776 2loss/FBt1/assignment_loss‚0.035677089275116994 1loss/SRqqt4/detection_loss‚1.7480605536928544ì;‚þÝžÎ"‚(:“ ñµåÔ³èó gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.633' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.526600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ÛÉ]!Ï"‚(:– ñµåÔõׇ( disk.nvme0c0n1.out‚40965.28466796875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 921.171875" proc.memory.rssMB‚ 2875.0078125 network.recv‚ 2.297378e+06 disk.sda3.out‚ 272.0859375# disk.dm-0.out‚40965.28466796875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.1323387e+07 disk.rbd1.out‚ 3.6796875 disk.sda2.out‚0 disk.rbd2.out‚188.3037109375 disk.rbd3.out‚0 disk.dm-0.in‚ 921.171875 disk.sda.in‚0 disk.dm-1.out‚ 272.0859375 disk.sda2.in‚0 disk.drbd1777.in‚0& disk.nvme0n1.out‚40965.28466796875 cpu‚25.3498334771288$ disk./.usageGB‚1100.816551208496 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0$ memory_percent‚94.19498443603516) proc.memory.percent‚17.54765510559082 disk.nvme0c0n1.in‚ 921.171875 disk.drbd1777.out‚0 disk.sda.out‚ 272.0859375% proc.memory.availableMB‚ 951.09375) disk./.usagePercent‚73.87672239820976$¬O‘Ð"‚(:† €¶åÔ›†Ú gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚99.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù”e‚!Ñ"‚(:– €¶åÔÜ…² disk.nvme0c0n1.in‚ 921.171875( disk.nvme0c0n1.out‚41064.35107421875 disk.nvme0n1.in‚ 921.171875 disk.rbd2.in‚0 disk.dm-0.in‚ 921.171875 cpu‚25.34918016225748 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0# disk.dm-0.out‚41064.35107421875 proc.cpu.threads‚17 network.recv‚ 2.302319e+06) disk./.usagePercent‚73.87672623832277 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0# memory_percent‚94.1951036453247) proc.memory.percent‚17.54765510559082 disk.sda1.in‚0 disk.dm-1.out‚ 272.578125 disk.rbd1.out‚ 3.6796875 disk.sda2.in‚0 network.sent‚ 2.1430568e+07 disk.drbd1777.out‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2875.0078125 disk.sda.in‚0 disk.sda.out‚ 272.578125& disk.nvme0n1.out‚41064.35107421875 disk.rbd1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 272.578125$ disk./.usageGB‚1100.816608428955 disk.rbd0.in‚0 disk.rbd2.out‚188.3818359375( proc.memory.availableMB‚ 951.07421875|6 6Ò"‚(:’ ¶åÔ«½„Ž gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.19' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.037999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÅWPÓ"‚(:Š ¶åÔ° # disk.dm-0.out‚41166.89111328125 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda3.out‚273.25 disk.drbd1777.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚1100.819637298584 disk.sda2.in‚0 disk.sda3.in‚0 network.recv‚ 2.305526e+06 disk.sda.in‚0& disk.nvme0n1.out‚41166.89111328125 disk.rbd1.out‚ 3.6796875) proc.memory.percent‚17.54765510559082 cpu‚25.34849718632828 disk.drbd1777.in‚0 disk.sda.out‚273.25 disk.sda1.out‚0 disk.dm-1.out‚273.25 disk.rbd0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2875.0078125 proc.cpu.threads‚17 network.sent‚ 2.143569e+07 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 921.171875 disk.sda2.out‚0 disk.rbd2.out‚188.3818359375$ memory_percent‚94.19596195220947( proc.memory.availableMB‚ 950.93359375 disk.rbd3.out‚0 disk.nvme0n1.in‚ 921.171875( disk.nvme0c0n1.out‚41166.89111328125) disk./.usagePercent‚73.87692950830497 disk.rbd3.in‚0 disk.dm-0.in‚ 921.171875ôaàz2Ô"- _wandbruntime‚6903  _runtime‚6903õæpàÉÕ"à ' lr-AdamW/pg2‚3.8454069636388434e-05 trainer/global_step‚9149 % _timestamp‚1.7884352184458814e+09 _runtime‚6903.150616358 _step‚749 ' lr-AdamW/pg1‚3.8454069636388434e-05ÁN0ÎÖ"È _step‚749 ' lr-AdamW/pg1‚3.8454069636388434e-05 ' lr-AdamW/pg2‚3.8454069636388434e-05 trainer/global_step‚9149 % _timestamp‚1.7884352184458814e+09 _runtime‚6903.150616358í€y02×"- _wandbruntime‚6903  _runtime‚6903´ûa’ðØ"àO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531889260645888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172715709 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 trainer/global_step‚9149 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8036722596352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚525586783944326 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43703757 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚580828103412224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580828079472128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172715709 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172695671 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚531889242642944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305715113 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121314921 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8036892511744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172695671 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121315376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7661947702784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚539550533983232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚539551208348672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚539550533983232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531889242642944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77611164 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43704210 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚531889260645888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7661947702784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440Ôdë'Ù"åO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7633086520586 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133019444 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚539551208348672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133019442 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77611166 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚525586112467210 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7633740958342 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚517953042985984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚539550533983232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚539551208348672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚539550533983232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531889242642944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77611164 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43704210 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚531889260645888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7661947702784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531889260645888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172715709 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 trainer/global_step‚9149 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8036722596352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚525586783944326 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43703757 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚580828103412224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580828079472128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172715709 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172695671 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚531889242642944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305715113 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121314921 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8036892511744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172695671 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121315376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7661947702784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305767649 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580889730209792 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121327530 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172734189 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 _step‚751 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133033458 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚539608380828672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚539607706449920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚531945595681792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172734189 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77619334 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7662767143936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133033460 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305747583 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚525641833032734 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚588927510211584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7662767143936 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305747583 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚580889754149888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77619332 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133033460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚518007930077184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚539608380828672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8037594548224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚9149 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172714125 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43708654 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚588927324758016 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚6903.824225638 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43708198 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚525642504510058 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7633902955550 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884352191189697e+09ò®?YÙ Ü"ðM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43708198 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚525642504510058 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7633902955550 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884352191189697e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7662110768128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531945595681792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305767649 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580889730209792 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121327530 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_en<Ÿúd/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172734189 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 _step‚751 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133033458 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚539608380828672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚539607706449920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚531945595681792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172734189 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77619334 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7662767143936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133033460 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305747583 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚525641833032734 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚588927510211584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7662767143936 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305747583 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚580889754149888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77619332 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133033460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚518007930077184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚539608380828672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8037594548224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚9149 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172714125 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43708654 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚588927324758016 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚6903.824225638 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752ïù#_T2Ý"- _wandbruntime‚6903  _runtime‚6903’³\U\Þ"Ö 0loss/FRt1/assignment_loss‚1.3312520958759166 0loss/FRt4/assignment_loss‚3.2373046927981908 /loss/FRt1/detection_loss‚1.4005002169697374 1loss/SRqqt3/detection_loss‚2.6637855619192123 trainer/global_step‚9149 % _timestamp‚1.7884352191209662e+09 2loss/SRqqt1/assignment_loss‚0.6683031626045703 2loss/SRqqt2/assignment_loss‚1.1252855834033755 _step‚752 2loss/SRqqt3/assignment_loss‚1.5526198016909452 0loss/SRqqt4/detection_loss‚2.022951820776576 epoch‚7 2loss/SRqqt4/assignment_loss‚1.5114605178435643 .loss/FRt2/detection_loss‚2.263726216797926 /loss/FRt4/detection_loss‚2.3059891124566394 .loss/FBt3/detection_loss‚3.849738041559855 0loss/FRt2/assignment_loss‚2.7479408991580105 /loss/FRt3/assignment_loss‚3.923655158135949 0loss/SRqqt1/detection_loss‚0.990178327386578 /loss/FBt1/detection_loss‚0.5964119632915901 %loss/total_loss‚42.90781520638788 1loss/FBt1/assignment_loss‚0.14143720174918947 1loss/SRqqt2/detection_loss‚2.1516848663489023 /loss/FBt2/detection_loss‚3.3158055543899536 1loss/FBt2/assignment_loss‚0.05033096469802937 2loss/FBt3/assignment_loss‚0.045269571244716644 1loss/FBt4/assignment_loss‚0.10232447289551298 .loss/FRt3/detection_loss‚2.690373980417484 _runtime‚ 6903.82561287 .loss/FBt4/detection_loss‚2.219485421975454“2aß"Û 0loss/FRt1/assignment_loss‚1.3312520958759166 0loss/FRt4/assignment_loss‚3.2373046927981908 /loss/FRt1/detection_loss‚1.4005002169697374 1loss/SRqqt3/detection_loss‚2.6637855619192123 trainer/global_step‚9149 % _timestamp‚1.7884352191209662e+09 2loss/SRqqt1/assignment_loss‚0.6683031626045703 2loss/SRqqt2/assignment_loss‚1.1252855834033755 _step‚752 2loss/SRqqt3/assignment_loss‚1.5526198016909452 0loss/SRqqt4/detection_loss‚2.022951820776576 epoch‚7 2loss/SRqqt4/assignment_loss‚1.5114605178435643 .loss/FRt2/detection_loss‚2.263726216797926 /loss/FRt4/detection_loss‚2.3059891124566394 .loss/FBt3/detection_loss‚3.849738041559855 0loss/FRt2/assignment_loss‚2.7479408991580105 /loss/FRt3/assignment_loss‚3.923655158135949 0loss/SRqqt1/detection_loss‚0.990178327386578 /loss/FBt1/detection_loss‚0.5964119632915901 %loss/total_loss‚42.90781520638788 1loss/FBt1/assignment_loss‚0.14143720174918947 1loss/SRqqt2/detection_loss‚2.1516848663489023 /loss/FBt2/detection_loss‚3.3158055543899536 1loss/FBt2/assignment_loss‚0.05033096469802937 2loss/FBt3/assignment_loss‚0.045269571244716644 1loss/FBt4/assignment_loss‚0.10232447289551298 .loss/FRt3/detection_loss‚2.690373980417484 _runtime‚ 6903.82561287 .loss/FBt4/detection_loss‚2.219485421975454ð­• à"‚(:’ ž¶åÔŽå gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.74' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.148000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸŽÛÿ%á"‚(:š ž¶åÔ„²œ disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 921.171875 network.recv‚ 2.30788e+06 disk.rbd2.in‚0 disk.sda.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚41167.37353515625 disk.rbd0.in‚0! disk.rbd2.out‚188.54638671875 disk.nvme0n1.in‚ 921.171875 disk.rbd1.out‚ 3.6796875% disk./.usageGB‚1100.8191947937012 disk.sda3.in‚0 disk.sda3.out‚ 273.73046875 disk.rbd3.out‚0# disk.dm-0.out‚41167.37353515625 disk.sda.out‚ 273.73046875 disk.nvme0c0n1.in‚ 921.171875" proc.memory.rssMB‚ 2875.0078125 disk.dm-1.out‚ 273.73046875) proc.memory.percent‚17.54765510559082 cpu‚25.347852346469622 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 network.sent‚ 2.1439461e+07' disk./.usagePercent‚73.876899811431 disk.sda2.in‚0 disk.sda1.out‚0$ memory_percent‚94.19918060302734 disk.sda1.in‚0& disk.nvme0n1.out‚41167.37353515625% proc.memory.availableMB‚ 950.40625ÙMZâ"‚(:’ ­¶åÔ¼¶ŸŽ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.310000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»‘ÞI!ã"‚(:– ­¶åÔ©Æï disk.sda2.out‚0 disk.drbd1777.in‚0$ memory_percent‚94.19910907745361) disk./.usagePercent‚73.87685449809743# disk.dm-0.out‚41267.14208984375& disk.nvme0n1.out‚41267.14208984375 disk.rbd2.in‚0 network.sent‚ 2.1546652e+07 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 921.171875 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.6796875 disk.rbd0.in‚0 disk.sda3.out‚ 274.53125( proc.memory.availableMB‚ 950.41796875) proc.memory.percent‚17.54765510559082 cpu‚25.34720507382671! disk.rbd2.out‚188.54638671875 disk.sda3.in‚0% disk./.usageGB‚1100.8185195922852 disk.rbd3.out‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚41267.14208984375" proc.memory.rssMB‚ 2875.0078125 proc.cpu.threads‚17 disk.dm-1.out‚ 274.53125 network.recv‚ 2.312185e+06 disk.dm-0.in‚ 921.171875 disk.sda.out‚ 274.53125 disk.nvme0n1.in‚ 921.171875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0âÂý2ä"- _wandbruntime‚6938  _runtime‚6938ŠP4HÇå"Á _step‚753 & lr-AdamW/pg1‚3.723391918348701e-05 & lr-AdamW/pg2‚3.723391918348701e-05 trainer/global_step‚9199 % _timestamp‚1.7884352537310555e+09 _runtime‚6938.435817428Dçb0Ìæ"Æ & lr-AdamW/pg1‚3.723391918348701e-05 & lr-AdamW/pg2‚3.723391918348701e-05 trainer/global_step‚9199 % _timestamp‚1.7884352537310555e+09 _runtime‚6938.435817428 _step‚753ñ–ç[·2ç"- _wandbruntime‚6938  _runtime‚6938È,‡æ'è"àO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7702277146112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚583910640295424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528372154334618 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133720242 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43931770 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚583910616355328 _runtime‚ 6938.43686978 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133720242 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121951334 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚534706894585344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173621723 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚470 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8079936586240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚534706912588288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8079764573696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚542409171731456 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚542409846096896 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚9199 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173641761 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚528372825811734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7673921859994 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7702277146112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542409171731456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7702933508608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43932240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚520698249513984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534706912588288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78019564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚542409846096896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173641761 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7674576297750 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _step‚754 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133720244 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7702933508608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚534706894585344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307341965 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78019566 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚591990576881664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚591990380929024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133720244 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121951806 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307362005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚520698232474624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542409171731456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7702933508608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43932240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚520698249513984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534706912588288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78019564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚542409846096896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173641761 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7674576297750 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _step‚754 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133720244 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7702933508608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚534706894585344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307341965 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78019566 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚591990576881664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚591990380929024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133720244 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121951806 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307362005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚520698232474624 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚534706912588288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8079764573696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚542409171731456 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401òÄwÛ2ê"- _wandbruntime‚6939  _runtime‚6939¸{#ïð&ë"êM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚542467018576896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307374435 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚528428546377466 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7703096573952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173640177 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121964024 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307394501 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7703096573952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133734258 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173640177 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚534763247624192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307394501 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚520753136605184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6939.105778102 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚583972267092992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚528427874900142 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133734260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚542467018576896 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚469 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7703752949760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚534763247624192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173660241 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78027734 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78027732 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7674738294958 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173660241 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚534763265627136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8080627004416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7675392732922 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43936759 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚583972291033088 _step‚755 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133734260 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542466344198144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133734258 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43936292 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788435254400342e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592052894097408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚520753153644544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 307374435 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592053081648128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚9199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121964493 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚534763265627136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624°ßÚõ&ì"ïM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121964493 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚534763265627136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚542467018576896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307374435 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚528428546377466 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7703096573952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173640177 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121964024 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307394501 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7703096573952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133734258 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173640177 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚534763247624192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307394501 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚520753136605184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6939.105778102 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚583972267092992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚528427874900142 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133734260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚542467018576896 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚469 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7703752949760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚534763247624192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173660241 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78027734 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78027732 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7674738294958 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173660241 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚534763265627136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8080627004416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7675392732922 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43936759 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚583972291033088 _step‚755 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133734260 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542466344198144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133734258 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43936292 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788435254400342e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592052894097408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚520753153644544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 307374435 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592053081648128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚9199ó?ÙÁ2í"- _wandbruntime‚6939  _runtime‚6939XÖ‡_î"Ù 0loss/FRt3/assignment_loss‚3.6247258434693017 2loss/SRqqt2/assignment_loss‚1.2057332906251153 trainer/global_step‚9199 _runtime‚6939.107187396 0loss/FRt4/assignment_loss‚2.7432985740403333 2loss/SRqqt1/assignment_loss‚0.8434642150609761 1loss/FBt2/assignment_loss‚0.20866427386257178 /loss/FRt1/detection_loss‚1.5456219104693292 /loss/FRt3/detection_loss‚3.1794565382103124 .loss/FRt4/detection_loss‚2.141059858724475 -loss/FRt2/detection_loss‚2.41139101361235 0loss/FRt1/assignment_loss‚1.0152658832834123 0loss/FRt2/assignment_loss‚2.5133848413825035 0loss/FBt1/detection_loss‚0.47777649098170516 % _timestamp‚1.7884352544025578e+09 2loss/SRqqt3/assignment_loss‚2.0613562585785985 /loss/FBt2/detection_loss‚2.1042483747005463 0loss/SRqqt4/detection_loss‚1.964043993841518 .loss/FBt4/detection_loss‚2.682771464188894 &loss/total_loss‚39.532304062112985 epoch‚7 1loss/FBt1/assignment_loss‚0.14342096885953454 0loss/SRqqt2/detection_loss‚1.651889114578565 _step‚756 0loss/FBt3/assignment_loss‚0.3032493202675444 2loss/FBt4/assignment_loss‚0.024807959406947095 1loss/SRqqt1/detection_loss‚0.9140260011190549 2loss/SRqqt4/assignment_loss‚1.3190441035072913 1loss/SRqqt3/detection_loss‚2.1991273313760757 /loss/FBt3/detection_loss‚2.2544764379660287U-dï"Þ 0loss/FRt3/assignment_loss‚3.6247258434693017 2loss/SRqqt2/assignment_loss‚1.2057332906251153 trainer/global_step‚9199 _runtime‚6939.107187396 0loss/FRt4/assignment_loss‚2.7432985740403333 2loss/SRqqt1/assignment_loss‚0.8434642150609761 1loss/FBt2/assignment_loss‚0.20866427386257178 /loss/FRt1/detection_loss‚1.5456219104693292 /loss/FRt3/detection_loss‚3.1794565382103124 .loss/FRt4/detection_loss‚2.141059858724475 -loss/FRt2/detection_loss‚2.41139101361235 0loss/FRt1/assignment_loss‚1.0152658832834123 0loss/FRt2/assignment_loss‚2.5133848413825035 0loss/FBt1/detection_loss‚0.47777649098170516 % _timestamp‚1.7884352544025578e+09 2loss/SRqqt3/assignment_loss‚2.0613562585785985 /loss/FBt2/detection_loss‚2.1042483747005463 0loss/SRqqt4/detection_loss‚1.964043993841518 .loss/FBt4/detection_loss‚2.682771464188894 &loss/total_loss‚39.532304062112985 epoch‚7 1loss/FBt1/assignment_loss‚0.14342096885953454 0loss/SRqqt2/detection_loss‚1.651889114578565 _step‚756 0loss/FBt3/assignment_loss‚0.3032493202675444 2loss/FBt4/assignment_loss‚0.024807959406947095 1loss/SRqqt1/detection_loss‚0.9140260011190549 2loss/SRqqt4/assignment_loss‚1.3190441035072913 1loss/SRqqt3/detection_loss‚2.1991273313760757 /loss/FBt3/detection_loss‚2.2544764379660287ôóü.žð"‚(:’ ¼¶åÔåÞƒŽ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚143.969' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.793799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú•ñ—*ñ"‚(:Ÿ ¼¶åÔ‚¶£‘ disk.dm-1.in‚0 disk.sda3.out‚ 274.86328125 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2875.0078125 cpu‚25.346562611790244 proc.cpu.threads‚17 network.sent‚ 2.1550533e+07 disk.dm-0.in‚ 921.171875 disk.sda1.in‚0 disk.rbd1.out‚ 3.6796875 disk.drbd1777.in‚0 disk.sda.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 950.6328125) proc.memory.percent‚17.54765510559082 network.recv‚ 2.314668e+06) disk./.usagePercent‚73.87698941406798 disk.nvme0n1.in‚ 921.171875& disk.nvme0n1.out‚41366.20458984375 disk.rbd1.in‚0( disk.nvme0c0n1.out‚41366.20458984375 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda2.out‚0! disk.rbd2.out‚190.13623046875# disk.dm-0.out‚41366.20458984375 disk.sda.out‚ 274.86328125 disk.dm-1.out‚ 274.86328125$ memory_percent‚94.19779777526855% disk./.usageGB‚1100.8205299377441 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 921.171875 disk.sda2.in‚0 disk.rbd3.out‚0 Xq·žò"‚(:“ ˶åÔŒñ€Ž gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.158' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.831599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÙŒh$ó"‚(:™ ˶åÔÜ‹å% disk./.usageGB‚1100.8203735351562 disk.sda.out‚ 275.47265625 disk.rbd1.out‚ 3.6796875 disk.sda3.out‚ 275.47265625 disk.dm-0.in‚ 921.171875 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 921.171875) proc.memory.percent‚17.54765510559082 disk.drbd1777.in‚0 network.sent‚ 2.1659499e+07 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0# memory_percent‚94.1992998123169 disk.nvme0n1.in‚ 921.171875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 950.38671875" proc.memory.rssMB‚ 2875.0078125 disk.sda.in‚0 network.recv‚ 2.321162e+06) disk./.usagePercent‚73.87697891775908! disk.dm-0.out‚41465.486328125 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0$ disk.nvme0n1.out‚41465.486328125& disk.nvme0c0n1.out‚41465.486328125 disk.sda2.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 275.47265625 disk.rbd1.in‚0! disk.rbd2.out‚190.13623046875 cpu‚25.345929429745173ÐÏH¶2ô"- _wandbruntime‚6973  _runtime‚6973ª'jÑÉõ"à ' lr-AdamW/pg1‚3.6029623164259835e-05 ' lr-AdamW/pg2‚3.6029623164259835e-05 trainer/global_step‚9249 % _timestamp‚1.7884352887473602e+09 _runtime‚6973.452098678 _step‚757³éÎö"È ' lr-AdamW/pg1‚3.6029623164259835e-05 ' lr-AdamW/pg2‚3.6029623164259835e-05 trainer/global_step‚9249 % _timestamp‚1.7884352887473602e+09 _runtime‚6973.452098678 _step‚757õ¤¸Ñü2÷"- _wandbruntime‚6973  _runtime‚6973ÿ~î¨Jø"áO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531158867679142 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚523443439002624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595116226022400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 trainer/global_step‚9249 _step‚758 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884352887481892e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134421044 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚6973.453253573 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚537524564530688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8122883122688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308988857 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚523443456041984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545267809479680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7743262951936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134421044 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚545268483845120 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134421042 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚545267809479680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚537524546527744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595116036361216 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78427966 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545268483845120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7743919314432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7743262951936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174547775 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586993177178624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8123048843776 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308968817 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134421042Ã$óÿì'ù"æO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7743919314432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44160063 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586993153238528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531158196202026 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚545267809479680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚537524546527744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595116036361216 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78427966 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545268483845120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7743919314432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7743262951936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174547775 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586993177178624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8123048843776 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308968817 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134421042 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531158867679142 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚523443439002624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595116226022400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 trainer/global_step‚9249 _step‚758 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884352887481892e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134421044 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚6973.453253573 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚537524564530688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8122883122688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308988857 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚523443456041984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545267809479680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7743262951936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134421044 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚545268483845120 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134421042 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587054803976192 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6974.112880278 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7744738755584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174586293 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174566229 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545324981946368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595178732804096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚523498360172544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚537580917569536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8123904887808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚587054827916288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134435058 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78436134 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309001287 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚523498343133184 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134435060 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134435058 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174586293 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚531213916767550 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚531214588244874 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7716228072330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7715573634366 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134435060 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309021353 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309001287 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174566229 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚537580917569536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 $ _timestamp‚1.788435289407562e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7744082379776 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122600214 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44164546 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚9249%ôœO2ü"ïM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚537580917569536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8123904887808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚587054827916288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134435058 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78436134 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309001287 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚523498343133184 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134435060 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134435058 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174586293 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚531213916767550 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚531214588244874 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7716228072330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7715573634366 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134435060 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309021353 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309001287 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174566229 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚537580917569536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 $ _timestamp‚1.788435289407562e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7744082379776 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122600214 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44164546 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚9249 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122600680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587054803976192 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6974.112880278 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F«†Ðà Monitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚537580899566592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7744082379776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8123741277184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7744738755584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚595178545253376 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7744738755584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174586293 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174566229 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545324981946368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595178732804096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚523498360172544÷í¾×ç2ý"- _wandbruntime‚6974  _runtime‚6974 }d.`þ"Ú 1loss/SRqqt3/detection_loss‚2.2872484111123614 1loss/SRqqt1/detection_loss‚0.8640668079024181 %loss/total_loss‚37.76255109365887 2loss/SRqqt3/assignment_loss‚2.0081033702525826 % _timestamp‚1.7884352894093606e+09 0loss/SRqqt4/detection_loss‚1.835711574388875 /loss/FRt1/detection_loss‚1.3090847891665274 /loss/FRt2/assignment_loss‚2.627633518186109 2loss/SRqqt4/assignment_loss‚1.2243819839010637 1loss/FBt1/assignment_loss‚0.07387276639684275 2loss/FBt2/assignment_loss‚0.036437566171536255 0loss/SRqqt2/detection_loss‚1.720331921892346 /loss/FBt1/detection_loss‚0.5600459452808233 /loss/FBt3/detection_loss‚2.7519951264063516 _step‚760 1loss/FBt3/assignment_loss‚0.14043957837081203 1loss/FBt4/assignment_loss‚0.12979747832287103 0loss/FRt1/assignment_loss‚1.1316433356654259 0loss/FRt3/assignment_loss‚2.5972849771380426 /loss/FBt4/detection_loss‚2.3142753168940544 /loss/FBt2/detection_loss‚2.3146238051928005 trainer/global_step‚9249 _runtime‚6974.113996351 epoch‚7 0loss/FRt4/assignment_loss‚2.4317725714372127 2loss/SRqqt2/assignment_loss‚1.4923745289023191 /loss/FRt4/detection_loss‚2.3534203439342734 1loss/SRqqt1/assignment_loss‚0.785294341185363 /loss/FRt2/detection_loss‚2.1770553945981224 /loss/FRt3/detection_loss‚2.5956556409597398T¬»ôeÿ"ß 1loss/SRqqt1/detection_loss‚0.8640668079024181 %loss/total_loss‚37.76255109365887 2loss/SRqqt3/assignment_loss‚2.0081033702525826 % _timestamp‚1.7884352894093606e+09 0loss/SRqqt4/detection_loss‚1.835711574388875 /loss/FRt1/detection_loss‚1.3090847891665274 /loss/FRt2/assignment_loss‚2.627633518186109 2loss/SRqqt4/assignment_loss‚1.2243819839010637 1loss/FBt1/assignment_loss‚0.07387276639684275 2loss/FBt2/assignment_loss‚0.036437566171536255 0loss/SRqqt2/detection_loss‚1.720331921892346 /loss/FBt1/detection_loss‚0.5600459452808233 /loss/FBt3/detection_loss‚2.7519951264063516 _step‚760 1loss/FBt3/assignment_loss‚0.14043957837081203 1loss/FBt4/assignment_loss‚0.12979747832287103 0loss/FRt1/assignment_loss‚1.1316433356654259 0loss/FRt3/assignment_loss‚2.5972849771380426 /loss/FBt4/detection_loss‚2.3142753168940544 /loss/FBt2/detection_loss‚2.3146238051928005 trainer/global_step‚9249 _runtime‚6974.113996351 epoch‚7 0loss/FRt4/assignment_loss‚2.4317725714372127 2loss/SRqqt2/assignment_loss‚1.4923745289023191 /loss/FRt4/detection_loss‚2.3534203439342734 1loss/SRqqt1/assignment_loss‚0.785294341185363 /loss/FRt2/detection_loss‚2.1770553945981224 /loss/FRt3/detection_loss‚2.5956556409597398 1loss/SRqqt3/detection_loss‚2.2872484111123614ø§Q!W‘€#‚(:† Ú¶åÔ›“â gpu.0.gpu‚41.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.96' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áQAò"#‚(:— Ú¶åԶʉ‘ disk.rbd1.out‚ 3.6796875 disk.rbd0.in‚0( proc.memory.availableMB‚ 950.58984375 cpu‚25.345257636950425 network.sent‚ 2.1664569e+07! disk.dm-0.out‚41465.779296875 disk.dm-1.out‚ 275.8671875 disk.sda3.in‚0 disk.sda1.in‚0 network.recv‚ 2.324397e+06 disk.dm-0.in‚ 921.171875 disk.nvme0c0n1.in‚ 921.171875 disk.rbd0.out‚0 disk.rbd2.in‚0$ memory_percent‚94.19806003570557% disk./.usageGB‚1100.8203773498535 disk.sda.out‚ 275.8671875 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0) disk./.usagePercent‚73.87697917376661 disk.rbd3.in‚0 disk.nvme0n1.in‚ 921.171875! disk.rbd2.out‚190.21435546875) proc.memory.percent‚17.54765510559082 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚41465.779296875 disk.sda3.out‚ 275.8671875 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.sda.in‚0$ disk.nvme0n1.out‚41465.779296875" proc.memory.rssMB‚ 2875.0078125‡ó ž‚#‚(:“ é¶åÔÞôâ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.463' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.692599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ޒσ#‚(:‡ é¶åÔŸ÷î‘ disk.drbd1777.in‚0 disk.sda3.out‚276.75' proc.memory.availableMB‚ 950.5703125 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚276.75& disk.nvme0c0n1.out‚41565.271484375 disk.sda2.out‚0 cpu‚25.344620386200507 network.sent‚ 2.1770858e+07 disk.nvme0n1.in‚ 921.171875 disk.rbd1.out‚ 3.6796875 proc.cpu.threads‚17 network.recv‚ 2.328612e+06$ disk.nvme0n1.out‚41565.271484375 disk.sda2.in‚0) disk./.usagePercent‚73.87697917376661 disk.rbd3.out‚0 disk.dm-0.in‚ 921.171875! disk.dm-0.out‚41565.271484375 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0! disk.rbd2.out‚190.21435546875% disk./.usageGB‚1100.8203773498535 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 921.171875 disk.rbd0.in‚0 disk.sda3.in‚0$ memory_percent‚94.19817924499512" proc.memory.rssMB‚ 2875.0078125) proc.memory.percent‚17.54765510559082 disk.sda.out‚276.75 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0ä«Ð¬“„#‚(:ˆ ø¶åÔË–ëŽ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚117.702' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5404 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûòy"…#‚(:— ø¶åÔ°³á‘& disk.nvme0n1.out‚41664.52978515625 disk.rbd1.out‚ 3.6796875" proc.memory.rssMB‚ 2875.0078125 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 921.171875 disk.rbd0.in‚0$ memory_percent‚94.19822692871094) proc.memory.percent‚17.54765510559082 disk.nvme0n1.in‚ 921.171875 network.sent‚ 2.1774673e+07 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 network.recv‚ 2.330464e+06 disk.rbd3.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚41664.52978515625 disk.sda.out‚ 277.43359375 disk.sda1.in‚0 cpu‚25.343992485243042 disk.dm-1.out‚ 277.43359375' disk./.usagePercent‚73.877015782844% disk./.usageGB‚1100.8209228515625 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 277.43359375# disk.dm-0.out‚41664.52978515625 disk.sda.in‚0 disk.rbd2.out‚ 190.31640625 disk.rbd3.in‚0 disk.sda1.out‚0$ proc.memory.availableMB‚950.5625 proc.cpu.threads‚17 disk.dm-0.in‚ 921.171875 Ä–22†#- _wandbruntime‚7008  _runtime‚7008\ð{uȇ# ' lr-AdamW/pg1‚3.4841432552192267e-05 ' lr-AdamW/pg2‚3.4841432552192267e-05 trainer/global_step‚9299 $ _timestamp‚1.788435323901093e+09 _runtime‚7008.605853769 _step‚761³êñ͈#Ç ' lr-AdamW/pg1‚3.4841432552192267e-05 ' lr-AdamW/pg2‚3.4841432552192267e-05 trainer/global_step‚9299 $ _timestamp‚1.788435323901093e+09 _runtime‚7008.605853769 _step‚761ùîh‰2‰#- _wandbruntime‚7008  _runtime‚70086&Gåç'Š#áO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44387745 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚598241752298496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175493865 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚540342198470144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7755592538810 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590075690121728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚540342198470144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135121842 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175493865 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175473827 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884353239018333e+09 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7784248757760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78836366 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78836364 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚548127121593344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚526188662569984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123224111 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚548126447227904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚533944238069434 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7784248757760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚598241564734464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8166038236672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135121842 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7784905120256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚548127121593344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7756246976566 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚9299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310595669 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚540342216473088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44387302 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚7008.606798439 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135121844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚540342216473088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚548126447227904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8165874612736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚762 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175473827 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680²ó‹êì'‹#æO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7784905120256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚762 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175473827 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44387745 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚598241752298496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175493865 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚540342198470144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7755592538810 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590075690121728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚540342198470144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135121842 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175493865 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175473827 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884353239018333e+09 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7784248757760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78836366 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78836364 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚548127121593344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚526188662569984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123224111 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚548126447227904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚533944238069434 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7784248757760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚598241564734464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8166038236672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135121842 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7784905120256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚548127121593344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7756246976566 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚9299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310595669 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚540342216473088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44387302 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚7008.606798439 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135121844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚540342216473088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚548126447227904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8165874612736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884353245634348e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590137340859392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚534000630112282 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175492281 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7785068185600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚548184294073344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚540398569511936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7785068185600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123236824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135135860 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7757063411738 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78844532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44392290 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310648205 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚526243566700544 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175512345 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135135858 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 310628139 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚533999958634958 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚7009.268763864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7785724561408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚540398551508992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598304084287488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚9299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310648205 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123236369 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8166743428096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135135860 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚548184294073344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590137364799488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚540398551508992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598304084287488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚9299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310648205 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123236369 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8166743428096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135135860 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚548184294073344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590137364799488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884353245634348e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590137340859392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚534000630112282 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175492281 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7785068185600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚548184294073344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚540398569511936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7785068185600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123236824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135135860 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7757063411738 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78844532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44392290 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310648205 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚526243566700544 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175512345 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135135858 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 310628139 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚533999958634958 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚7009.268763864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752ûÖI2#- _wandbruntime‚7009  _runtime‚7009|“ú)\#Ö 2loss/SRqqt4/assignment_loss‚1.6514693640594214 /loss/FBt1/detection_loss‚0.7985144334028743 2loss/SRqqt2/assignment_loss‚1.2364355157284688 2loss/SRqqt3/assignment_loss‚1.7502217753324658 1loss/FBt4/assignment_loss‚0.13608365869996222 0loss/SRqqt1/detection_loss‚1.016767262835318 1loss/SRqqt4/detection_loss‚1.8204229161986765 0loss/FRt2/assignment_loss‚2.9962981026619673 2loss/SRqqt1/assignment_loss‚0.9941461267618751 0loss/FBt2/assignment_loss‚0.2151351167509953 % _timestamp‚1.7884353245651271e+09 /loss/FRt4/assignment_loss‚3.312259566783905 1loss/FBt3/assignment_loss‚0.18242077013322464 /loss/FRt4/detection_loss‚2.2174754977226256 0loss/SRqqt2/detection_loss‚1.863756386252741 %loss/total_loss‚44.77261791937732 1loss/FBt1/assignment_loss‚0.13551517955652673 /loss/FRt2/detection_loss‚2.9503544718027115 .loss/FBt2/detection_loss‚2.417413334051768 .loss/FBt4/detection_loss‚2.229881102388555 _runtime‚7009.270126241 0loss/FRt1/assignment_loss‚1.9666298002493185 0loss/FRt3/assignment_loss‚3.8938174431140604 /loss/FRt3/detection_loss‚3.0238765753232517 1loss/SRqqt3/detection_loss‚2.3556477442383765 .loss/FBt3/detection_loss‚4.173452377319336 trainer/global_step‚9299 /loss/FRt1/detection_loss‚1.4346233980088938 epoch‚7 _step‚7641ùv&a‘#Û 1loss/SRqqt4/detection_loss‚1.8204229161986765 0loss/FRt2/assignment_loss‚2.9962981026619673 2loss/SRqqt1/assignment_loss‚0.9941461267618751 0loss/FBt2/assignment_loss‚0.2151351167509953 % _timestamp‚1.7884353245651271e+09 /loss/FRt4/assignment_loss‚3.312259566783905 1loss/FBt3/assignment_loss‚0.18242077013322464 /loss/FRt4/detection_loss‚2.2174754977226256 0loss/SRqqt2/detection_loss‚1.863756386252741 %loss/total_loss‚44.77261791937732 1loss/FBt1/assignment_loss‚0.13551517955652673 /loss/FRt2/detection_loss‚2.9503544718027115 .loss/FBt2/detection_loss‚2.417413334051768 .loss/FBt4/detection_loss‚2.229881102388555 _runtime‚7009.270126241 0loss/FRt1/assignment_loss‚1.9666298002493185 0loss/FRt3/assignment_loss‚3.8938174431140604 /loss/FRt3/detection_loss‚3.0238765753232517 1loss/SRqqt3/detection_loss‚2.3556477442383765 .loss/FBt3/detection_loss‚4.173452377319336 trainer/global_step‚9299 /loss/FRt1/detection_loss‚1.4346233980088938 epoch‚7 _step‚764 2loss/SRqqt4/assignment_loss‚1.6514693640594214 /loss/FBt1/detection_loss‚0.7985144334028743 2loss/SRqqt2/assignment_loss‚1.2364355157284688 2loss/SRqqt3/assignment_loss‚1.7502217753324658 1loss/FBt4/assignment_loss‚0.13608365869996222 0loss/SRqqt1/detection_loss‚1.016767262835318üÿóH“’#‚(:ˆ ‡·åÔóÒˆŽ gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.399' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.2798 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š>j“(“#‚(: ‡·åÔÕÔþ‘) disk./.usagePercent‚73.87701219873853 disk.rbd1.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 935.5390625% disk.nvme0n1.out‚41764.4345703125 disk.sda2.in‚0 disk.rbd2.out‚ 190.31640625 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda.out‚ 277.94921875 proc.cpu.threads‚17 network.recv‚ 2.333371e+06 disk.rbd1.out‚ 3.6796875! disk.nvme0c0n1.in‚ 935.5390625' disk.nvme0c0n1.out‚41764.4345703125 disk.sda2.out‚0 disk.sda3.out‚ 277.94921875 disk.rbd3.in‚0 disk.drbd1777.in‚0 disk.sda1.out‚0$ memory_percent‚94.20125484466553 network.sent‚ 2.1779294e+07% disk./.usageGB‚1100.8208694458008 disk.rbd0.out‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2875.0078125 disk.dm-1.out‚ 277.94921875 disk.drbd1777.out‚0" disk.dm-0.out‚41764.4345703125 disk.sda1.in‚0( proc.memory.availableMB‚ 950.06640625 cpu‚25.343365071484655 disk.rbd0.in‚0 disk.dm-0.in‚ 935.5390625) proc.memory.percent‚17.54765510559082Ãà ¬ž”#‚(:“ –·åÔ¾ºˆŽ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.759' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.551799999999993 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙùDm+•#‚(:  –·åÔ‰ð… network.sent‚ 2.1887444e+07! disk.nvme0n1.in‚ 1067.51953125 disk.rbd2.in‚0 disk.rbd2.out‚ 191.90625 disk.sda3.in‚0 disk.sda1.in‚0 network.recv‚ 2.338857e+06 disk.sda.in‚0 disk.sda.out‚ 278.30078125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚73.87701091870086 disk.drbd1777.in‚0" disk.dm-0.out‚41863.5205078125' disk.nvme0c0n1.out‚41863.5205078125" proc.memory.rssMB‚ 2875.0078125) proc.memory.percent‚17.54765510559082 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 950.53515625% disk./.usageGB‚1100.8208503723145 disk.dm-0.in‚ 1067.51953125% disk.nvme0n1.out‚41863.5205078125 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 278.30078125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875# disk.nvme0c0n1.in‚ 1067.51953125 disk.sda3.out‚ 278.30078125$ memory_percent‚94.19839382171631 cpu‚25.3427209069992869›y2–#- _wandbruntime‚7043  _runtime‚7043 F|Ú×#½ $ lr-AdamW/pg1‚3.3669594964427e-05 $ lr-AdamW/pg2‚3.3669594964427e-05 trainer/global_step‚9349 % _timestamp‚1.7884353591878111e+09 _runtime‚7043.892508276 _step‚7652ÄÝÖȘ# % _timestamp‚1.7884353591878111e+09 _runtime‚7043.892508276 _step‚765 $ lr-AdamW/pg1‚3.3669594964427e-05 $ lr-AdamW/pg2‚3.3669594964427e-05 trainer/global_step‚9349ýÜ7‘Â2™#- _wandbruntime‚7043  _runtime‚7043W³Ä½!š#áO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚543159868415488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135822644 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚536730279936842 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚528933869097984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176419917 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚550985084976128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚543159850412544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176399879 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚550985759341568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44615428 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7797082315974 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7825890926080 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚766 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚550985084976128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7825890926080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44614981 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚7043.893395257 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79244766 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176419917 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123860194 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593158250945024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Ä…ì'›#æO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚528933852058624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312222521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚550985759341568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44615428 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7797082315974 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7825890926080 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚766 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚550985084976128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7825890926080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44614981 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚7043.893395257 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79244766 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176419917 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123860194 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593158250945024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚543159868415488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135822644 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚536730279936842 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚528933869097984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176419917 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚550985084976128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚543159850412544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176399879 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123872432 _step‚767 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176418333 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312275057 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7826053991424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312275057 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7826710367232 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135836658 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528988756189184 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚543216221454336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚551042931821568 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 _runtime‚7044.570311019 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7826710367232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44619500 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593219877742592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8209963412480 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚551042931821568 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312254991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79252934 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176418333 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚551042257442816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7797244313182 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884353598651721e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44619939 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚9349 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7826053991424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79252932 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601429683835904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚593219901682688 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123872432 _step‚767 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176418333 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312275057 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7826053991424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312275057 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7826710367232 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135836658 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528988756189184 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚543216221454336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚551042931821568 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 _runtime‚7044.570311019 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7826710367232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44619500 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593219877742592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8209963412480 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚551042931821568 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312254991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79252934 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176418333 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚551042257442816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7797244313182 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884353598651721e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44619939 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚9349 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7826053991424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79252932 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601429683835904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚593219901682688 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚553843722724352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539516321804250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124496666 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545977502354944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8251987447296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚604492930510848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚553844397089792 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7866876731904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7837263217626 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7866876731904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44843068 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚596240787828224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177345969 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79653164 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136523444 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136523442 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79653166 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚604492751335424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚596240763888128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545977520357888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136523442 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177345969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44843500 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884353950589027e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539516993281366 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚531679058586624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚553844397089792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553843722724352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124496232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7866220369408 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7866220369408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _step‚770 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313849373 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313869413 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8252142682624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚545977502354944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136523444 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 _runtime‚ 7079.7637657 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7837917655382 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177325931 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚545977520357888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79653164 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136523444 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136523442 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79653166 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚604492751335424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚596240763888128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545977520357888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136523442 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177345969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44843500 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884353950589027e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539516993281366 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚531679058586624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚553844397089792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553843722724352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124496232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7866220369408 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7866220369408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _step‚770 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313849373 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313869413 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8252142682624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚545977502354944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136523444 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 _runtime‚ 7079.7637657 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7837917655382 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177325931 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚545977520357888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚553843722724352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539516321804250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124496666 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545977502354944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8251987447296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚604492930510848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430‚ ‘+¬#‚(:’ ÷åÔµöêŽ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.19' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.037999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[6N¦­#‚(:Š ÷åÔ·Ž» disk.drbd1777.in‚0 disk.dm-0.out‚ 42062.65625! disk.nvme0n1.in‚ 1067.51953125' proc.memory.availableMB‚ 950.2421875 network.recv‚ 2.348833e+06 disk.drbd1777.out‚0 disk.dm-0.in‚ 1067.51953125 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.out‚280.1875 disk.rbd1.out‚ 3.6796875 disk.rbd0.out‚0) proc.memory.percent‚17.54765510559082 cpu‚25.340899284998592 proc.cpu.threads‚17) disk./.usagePercent‚73.87701680687414 disk.rbd3.in‚0 disk.sda.out‚280.1875 disk.rbd2.in‚0$ memory_percent‚94.20018196105957 network.sent‚ 2.2002523e+07 disk.rbd3.out‚0 disk.sda1.in‚0" disk.nvme0c0n1.out‚ 42062.65625 disk.sda3.out‚280.1875 disk.sda.in‚0 disk.nvme0n1.out‚ 42062.65625# disk.nvme0c0n1.in‚ 1067.51953125 disk.rbd0.in‚0! disk.rbd2.out‚192.07080078125% disk./.usageGB‚1100.8209381103516 disk.sda2.out‚0" proc.memory.rssMB‚ 2875.0078125PÊT2®#- _wandbruntime‚7080  _runtime‚7080¥ûßoj¯#êM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7867039797248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚546033855393792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313881843 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136537460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177364449 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539572713847098 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44847562 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚7080.431379575 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136537460 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚596302414625792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8252868174336 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚604555282800128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.alloßT¬»†cated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313901909 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7838734090554 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553900895191040 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313881843 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124508894 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚531733962717184 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177364449 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7867696173056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚546033873396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79661334 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚9399 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313901909 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79661332 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.788435395726191e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8253021299200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚546033855393792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553901569569792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124509336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177344385 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124508894 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚531733962717184 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177364449 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7867696173056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚546033873396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79661334 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚9399 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313901909 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79661332 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.788435395726191e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8253021299200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚546033855393792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553901569569792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124509336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177344385 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚604555282800128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313901909 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7838734090554ƒȺf$2±#- _wandbruntime‚7080  _runtime‚7080 l[²#Õ /loss/FRt2/assignment_loss‚2.644316183357704 1loss/FBt2/assignment_loss‚0.15014003313262947 /loss/FBt1/detection_loss‚0.9187463063893985 .loss/FBt3/detection_loss‚2.615718492439815 /loss/FBt4/detection_loss‚2.3059182862440744 0loss/FRt3/assignment_loss‚3.5688431806424084 0loss/FBt4/assignment_loss‚0.8051532243068019 .loss/FRt2/detection_loss‚2.394056952581173 /loss/FRt4/detection_loss‚2.2529622337152793 trainer/global_step‚9399 _step‚772 0loss/FRt4/assignment_loss‚3.0456196249917498 1loss/SRqqt3/assignment_loss‚1.864074088824101 $loss/total_loss‚41.9127774036861 _runtime‚7080.432346126 2loss/SRqqt1/assignment_loss‚0.9275634898700471 2loss/SRqqt2/assignment_loss‚1.2902480914360948 2loss/SRqqt4/assignment_loss‚1.4207943064953994 % _timestamp‚1.7884353957277374e+09 0loss/FRt1/assignment_loss‚1.3483697610596816 .loss/FRt1/detection_loss‚1.401189602445811 0loss/SRqqt3/detection_loss‚2.222075719099778 1loss/FBt1/assignment_loss‚0.14863437467168844 epoch‚7 1loss/FBt3/assignment_loss‚0.12300821875994318 /loss/FRt3/detection_loss‚2.8097670253585365 1loss/SRqqt1/detection_loss‚0.9705095233472243 1loss/SRqqt2/detection_loss‚1.6701518732403953 1loss/SRqqt4/detection_loss‚1.9465103486069926 /loss/FBt2/detection_loss‚3.0684064626693726ër`³#Ú 0loss/SRqqt3/detection_loss‚2.222075719099778 1loss/FBt1/assignment_loss‚0.14863437467168844 epoch‚7 1loss/FBt3/assignment_loss‚0.12300821875994318 /loss/FRt3/detection_loss‚2.8097670253585365 1loss/SRqqt1/detection_loss‚0.9705095233472243 1loss/SRqqt2/detection_loss‚1.6701518732403953 1loss/SRqqt4/detection_loss‚1.9465103486069926 /loss/FBt2/detection_loss‚3.0684064626693726 /loss/FRt2/assignment_loss‚2.644316183357704 1loss/FBt2/assignment_loss‚0.15014003313262947 /loss/FBt1/detection_loss‚0.9187463063893985 .loss/FBt3/detection_loss‚2.615718492439815 /loss/FBt4/detection_loss‚2.3059182862440744 0loss/FRt3/assignment_loss‚3.5688431806424084 0loss/FBt4/assignment_loss‚0.8051532243068019 .loss/FRt2/detection_loss‚2.394056952581173 /loss/FRt4/detection_loss‚2.2529622337152793 trainer/global_step‚9399 _step‚772 0loss/FRt4/assignment_loss‚3.0456196249917498 1loss/SRqqt3/assignment_loss‚1.864074088824101 $loss/total_loss‚41.9127774036861 _runtime‚7080.432346126 2loss/SRqqt1/assignment_loss‚0.9275634898700471 2loss/SRqqt2/assignment_loss‚1.2902480914360948 2loss/SRqqt4/assignment_loss‚1.4207943064953994 % _timestamp‚1.7884353957277374e+09 0loss/FRt1/assignment_loss‚1.3483697610596816 .loss/FRt1/detection_loss‚1.401189602445811„ûe ýž´#‚(:“ Ò·åÔó늎 gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.463' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.492599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë‘)µ#‚(:ž Ò·åÔìôþ‘ network.recv‚ 2.353646e+06 disk.dm-0.out‚42161.80859375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 280.62109375 disk.rbd3.out‚0 disk.sda.out‚ 280.62109375 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0' proc.memory.availableMB‚ 950.4453125! disk.nvme0n1.in‚ 1067.51953125# disk.nvme0c0n1.in‚ 1067.51953125$ memory_percent‚94.19894218444824) proc.memory.percent‚17.54765510559082 proc.cpu.threads‚17 network.sent‚ 2.2098408e+07) disk./.usagePercent‚73.87701987896456% disk./.usageGB‚1100.8209838867188 disk.dm-1.in‚0 disk.rbd0.out‚0! disk.rbd2.out‚192.14501953125 disk.rbd3.in‚0 disk.dm-1.out‚ 280.62109375 disk.rbd1.in‚0% disk.nvme0c0n1.out‚42161.80859375 disk.sda1.in‚0# disk.nvme0n1.out‚42161.80859375 disk.sda2.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 1067.51953125 disk.sda.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.6796875" proc.memory.rssMB‚ 2875.0078125 cpu‚25.34027524849698†ý̶#‚(:’ á·åÔ²ú•Ž gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.766000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Dê02·#‚(:§ á·åÔ¯÷ã disk.drbd1777.out‚0# disk.dm-0.out‚42162.86865234375 disk.sda1.in‚0" proc.memory.rssMB‚ 2875.0078125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1067.51953125 disk.sda1.out‚0 disk.dm-1.out‚ 281.16796875 network.recv‚ 2.360237e+06! disk.rbd2.out‚193.65673828125 disk.dm-1.in‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1067.51953125( disk.nvme0c0n1.out‚42162.86865234375 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 2.2196638e+07 disk.rbd1.out‚ 3.6796875' proc.memory.availableMB‚ 950.1953125& disk.nvme0n1.out‚42162.86865234375 disk.sda3.out‚ 281.16796875$ memory_percent‚94.20046806335449! disk.nvme0n1.in‚ 1067.51953125) disk./.usagePercent‚73.87701962295702% disk./.usageGB‚1100.8209800720215 disk.drbd1777.in‚0 disk.sda.in‚0) proc.memory.percent‚17.54765510559082 disk.rbd0.out‚0 disk.sda.out‚ 281.16796875 cpu‚25.33967395469151ã9,2¸#- _wandbruntime‚7115  _runtime‚7115£`ÉÖǹ#Á _runtime‚7115.347241939 _step‚773 & lr-AdamW/pg1‚3.137595223975216e-05 & lr-AdamW/pg2‚3.137595223975216e-05 trainer/global_step‚9449 % _timestamp‚1.7884354306424518e+09Ú Á#̺#Æ _runtime‚7115.347241939 _step‚773 & lr-AdamW/pg1‚3.137595223975216e-05 & lr-AdamW/pg2‚3.137595223975216e-05 trainer/global_step‚9449 % _timestamp‚1.7884354306424518e+09…w” f2»#- _wandbruntime‚7115  _runtime‚7115~0Sëç'¼#áO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚9449 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚548795172300288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚548795154297344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542303035148774 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8295172168704 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125134047 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7907862537728 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7907862537728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7907206175232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80061564 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45072037 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚556702360472576 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137224244 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125133601 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚542302363671658 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚599323324711424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚7115.348163796 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137224242 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚548795172300288 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178272021 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137224244 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178251983 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178251983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80061566 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7878752994790 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚556703034838016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884354306432269e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 315496265 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚534424265114624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚556703034838016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8295331598336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137224242 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚599323300771328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7878098557034 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315476225 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178272021 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7907206175232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚548795154297344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45072481 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556702360472576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚607618656309760 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7878752994790 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚556703034838016 LFDeviceStatsMonitor.on_trainÃG1"_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884354306432269e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 315496265 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚534424265114624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚556703034838016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8295331598336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137224242 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚599323300771328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7878098557034 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315476225 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178272021 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7907206175232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚548795154297344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45072481 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556702360472576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚607618656309760 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚9449 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚548795172300288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚548795154297344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542303035148774 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8295172168704 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125134047 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7907862537728 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7907862537728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7907206175232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80061564 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45072037 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚556702360472576 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137224244 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125133601 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚542302363671658 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚599323324711424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚7115.348163796 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137224242 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚548795172300288 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844†›«2¾#- _wandbruntime‚7116  _runtime‚7116¹z’bð&¿#êM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚599384975449088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚542358084237182 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178290501 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315508695 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178290501 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚775 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7908025603072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137238258 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80069734 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178270437 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚548851525339136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137238260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315528761 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125146766 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45076591 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884354313200052e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137238260 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚556760207318016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315508695 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚556759532939264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚607681176449536 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚534479169245184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80069732 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7908025603072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125146323 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚ 7116.02539086 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542358755714506 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137238258 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178270437 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚599384951508992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚548851507336192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607680997287424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚9449 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315528761 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45077032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8296201000448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80069732 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7908025603072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125146323 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚ 7116.02539086 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542358755714506 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137238258 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178270437 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚599384951508992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚548851507336192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607680997287424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚9449 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315528761 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45077032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8296201000448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7908025603072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137238258 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80069734 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178270437 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚548851525339136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137238260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315528761 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125146766 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45076591 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884354313200052e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137238260 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚556760207318016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315508695 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚556759532939264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚607681176449536 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚534479169245184‡¤ûI2Á#- _wandbruntime‚7116  _runtime‚7116¸VÂ#Ð 2loss/SRqqt1/assignment_loss‚1.1144758926606493 1loss/SRqqt4/assignment_loss‚1.268669025399288 0loss/SRqqt3/detection_loss‚2.398835876892353 _step‚776 1loss/SRqqt2/assignment_loss‚1.192649065545283 0loss/FRt3/assignment_loss‚3.2379210513571035 1loss/FBt1/assignment_loss‚0.10263529808814356 1loss/FBt3/assignment_loss‚0.23374420792485276 % _timestamp‚1.7884354313218777e+09 0loss/FRt1/assignment_loss‚1.1230426881048414 trainer/global_step‚9449 .loss/FBt4/detection_loss‚2.187674038940006 .loss/FRt2/assignment_loss‚2.67866537460061 1loss/SRqqt3/assignment_loss‚1.485657248549677 1loss/FBt2/assignment_loss‚0.07977464443279637 1loss/FBt4/assignment_loss‚0.25019974370176595 .loss/FRt2/detection_loss‚2.452831650889197 0loss/SRqqt4/detection_loss‚1.558595094135252 epoch‚7 /loss/FRt4/detection_loss‚1.9607046826018228 1loss/SRqqt1/detection_loss‚1.0150070027039761 0loss/SRqqt2/detection_loss‚1.900195003646176 /loss/FBt3/detection_loss‚2.3520735104878745 _runtime‚7116.026831354 -loss/FRt1/detection_loss‚1.60068128514621 /loss/FRt3/detection_loss‚3.3583832383155823 /loss/FBt1/detection_loss‚0.7007687609424214 .loss/FBt2/detection_loss‚3.115083555380503 &loss/total_loss‚40.103412601978896 /loss/FRt4/assignment_loss‚2.735144661532508¹í¤[Ã#Õ 1loss/FBt3/assignment_loss‚0.23374420792485276 % _timestamp‚1.7884354313218777e+09 0loss/FRt1/assignment_loss‚1.1230426881048414 trainer/global_step‚9449 .loss/FBt4/detection_loss‚2.187674038940006 .loss/FRt2/assignment_loss‚2.67866537460061 1loss/SRqqt3/assignment_loss‚1.485657248549677 1loss/FBt2/assignment_loss‚0.07977464443279637 1loss/FBt4/assignment_loss‚0.25019974370176595 .loss/FRt2/detection_loss‚2.452831650889197 0loss/SRqqt4/detection_loss‚1.558595094135252 epoch‚7 /loss/FRt4/detection_loss‚1.9607046826018228 1loss/SRqqt1/detection_loss‚1.0150070027039761 0loss/SRqqt2/detection_loss‚1.900195003646176 /loss/FBt3/detection_loss‚2.3520735104878745 _runtime‚7116.026831354 -loss/FRt1/detection_loss‚1.60068128514621 /loss/FRt3/detection_loss‚3.3583832383155823 /loss/FBt1/detection_loss‚0.7007687609424214 .loss/FBt2/detection_loss‚3.115083555380503 &loss/total_loss‚40.103412601978896 /loss/FRt4/assignment_loss‚2.735144661532508 2loss/SRqqt1/assignment_loss‚1.1144758926606493 1loss/SRqqt4/assignment_loss‚1.268669025399288 0loss/SRqqt3/detection_loss‚2.398835876892353 _step‚776 1loss/SRqqt2/assignment_loss‚1.192649065545283 0loss/FRt3/assignment_loss‚3.2379210513571035 1loss/FBt1/assignment_loss‚0.10263529808814356ˆÂ[ Ä#‚(:’ ð·åÔ•ïŽ gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚88.126' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.625200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«@)´ùÅ#‚(:¢ ð·åԉɰ‘ disk.drbd1777.out‚0# disk.dm-0.out‚42261.90771484375 disk.dm-1.in‚0! disk.nvme0n1.in‚ 1067.51953125& disk.nvme0n1.out‚42261.90771484375 disk.sda3.out‚ 281.921875 disk.rbd3.out‚0( disk.nvme0c0n1.out‚42261.90771484375 disk.sda2.in‚0( proc.memory.availableMB‚ 950.42578125 proc.cpu.threads‚17% disk./.usageGB‚1100.8210372924805 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 1067.51953125 disk.sda1.in‚0! disk.rbd2.out‚193.79736328125 disk.rbd3.in‚0 disk.rbd1.out‚ 3.6796875" proc.memory.rssMB‚ 2875.0078125 cpu‚25.339062027268266 network.recv‚ 2.362141e+06 disk.sda.in‚0 diskP…ß4.sda.out‚ 281.921875 disk.dm-1.out‚ 281.921875 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1067.51953125# memory_percent‚94.1990613937378 network.sent‚ 2.2200152e+07) disk./.usagePercent‚73.87702346307003 disk.drbd1777.in‚0 disk.sda2.out‚0) proc.memory.percent‚17.54765510559082‚‘Æ#‚(:† ÿ·åÔ—æé gpu.0.gpu‚2.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.593' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.5186 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê/<¼-Ç#‚(:¢ ÿ·åÔ—ë‚‘ disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 282.5546875 disk.sda2.in‚0( proc.memory.availableMB‚ 950.51171875" proc.memory.rssMB‚ 2875.0078125 network.recv‚ 2.368048e+06 disk.dm-0.in‚ 1067.51953125 disk.sda1.in‚0% disk.nvme0n1.out‚42361.7783203125 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 3.6796875 cpu‚25.33849447677047% disk./.usageGB‚1100.8209800720215" disk.dm-0.out‚42361.7783203125' disk.nvme0c0n1.out‚42361.7783203125 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 1067.51953125# disk.nvme0c0n1.in‚ 1067.51953125$ memory_percent‚94.19853687286377 network.sent‚ 2.2308727e+07) disk./.usagePercent‚73.87701962295702 disk.rbd0.out‚0! disk.rbd2.out‚193.79736328125 disk.sda.in‚0) proc.memory.percent‚17.54765510559082 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 282.5546875 disk.dm-1.out‚ 282.5546875 disk.rbd1.in‚0 proc.cpu.threads‚17 ñn2È#- _wandbruntime‚7151  _runtime‚7151«!—ÆÉ#À _runtime‚7151.237777883 _step‚777 & lr-AdamW/pg1‚3.025462509454866e-05 & lr-AdamW/pg2‚3.025462509454866e-05 trainer/global_step‚9499 $ _timestamp‚1.788435466533011e+09!ëSËÊ#Å & lr-AdamW/pg1‚3.025462509454866e-05 & lr-AdamW/pg2‚3.025462509454866e-05 trainer/global_step‚9499 $ _timestamp‚1.788435466533011e+09 _runtime‚7151.237777883 _step‚777‰ych$2Ë#- _wandbruntime‚7151  _runtime‚7151’§Øƒæ'Ì#àO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45301730 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610744108390912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚551612806239744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151041024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 $ _timestamp‚1.788435466533762e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137925042 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7948191981056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7948191981056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80469966 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602405837654528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚551612824242688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚551612806239744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚537169471642624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚537169488681984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125771257 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚610744283372032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7918933896442 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137925044 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545088405539066 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559561672586240 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚559560998220800 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚7151.238748591 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137925042 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179198073 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7948848343552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179178035 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559560998220800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179178035 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45301293 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317103077 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179198073 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚551612824242688 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7948848343552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908´°Áë'Í#åO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602405837654528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚551612824242688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚551612806239744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚537169471642624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚537169488681984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125771257 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚610744283372032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7918933896442 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137925044 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545088405539066 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559561672586240 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚559560998220800 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚7151.238748591 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137925042 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179198073 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7948848343552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179178035 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559560998220800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179178035 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45301293 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317103077 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179198073 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚551612824242688 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7948848343552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45301730 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610744108390912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚551612806239744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151041024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 $ _timestamp‚1.788435466533762e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137925042 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7948191981056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7948191981056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80469966 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376ŠåúÃû2Î#- _wandbruntime‚7151  _runtime‚7151àH#É&Ï#ëM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559618170687488 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559618845066240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7949667784704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚779 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317155613 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137939060 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚602467488392192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚551669177281536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45306250 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317155613 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45305806 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179196489 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8339121777152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚9499 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚559618845066240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚551669159278592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7919750331406 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚551669177281536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80478132 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179196489 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚551669159278592 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137939060 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7949667784704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137939058 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8339289582080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545144797581914 _runtime‚7151.906388438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7920404769370 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884354672011988e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179216553 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚610806610169344 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7949011408896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537224392812544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179216553 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125783938 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 QCDeviceStatsMonitor.on_train_batch_end/iª(nactive_split_bytes.all.peak‚ 337538560ƒÁÔHö&Ð#ðM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45306250 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317155613 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45305806 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179196489 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8339121777152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚9499 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚559618845066240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚551669159278592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7919750331406 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚551669177281536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80478132 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179196489 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚551669159278592 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137939060 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7949667784704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137939058 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8339289582080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545144797581914 _runtime‚7151.906388438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7920404769370 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884354672011988e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179216553 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚610806610169344 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7949011408896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537224392812544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179216553 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125783938 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559618170687488 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559618845066240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7949667784704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚779 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317155613 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137939060 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚602467488392192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚551669177281536‹}ŠñØ2Ñ#- _wandbruntime‚7151  _runtime‚7151ºµïYÒ#Ó 2loss/SRqqt2/assignment_loss‚1.3851848445259607 1loss/SRqqt3/assignment_loss‚1.831579381806983 1loss/FBt4/assignment_loss‚0.07959052852432554 0loss/FRt3/assignment_loss‚2.9170303811898104 /loss/FBt3/detection_loss‚3.0376606583595276 0loss/FRt1/assignment_loss‚1.5751725914685621 /loss/FRt3/detection_loss‚2.6388706355481535 %loss/total_loss‚40.49463730163612 /loss/FBt1/assignment_loss‚0.141345983504921 .loss/FBt2/detection_loss‚2.791399821639061 % _timestamp‚1.7884354672027893e+09 /loss/FRt2/assignment_loss‚2.215552420150943 2loss/SRqqt1/assignment_loss‚0.8040420058387099 /loss/FBt4/detection_loss‚2.5221563246515064 epoch‚7 trainer/global_step‚9499 0loss/SRqqt2/detection_loss‚1.489489731651086 0loss/SRqqt4/detection_loss‚1.995889535292666 1loss/FBt2/assignment_loss‚0.17162520197598496 2loss/FBt3/assignment_loss‚0.027227828948525712 /loss/FRt1/detection_loss‚1.7624319607958845 .loss/FRt2/detection_loss‚2.430091532265268 /loss/FRt4/detection_loss‚2.3638091557051824 1loss/SRqqt3/detection_loss‚2.3432241529226303 /loss/FBt1/detection_loss‚0.5982853825244516 _runtime‚7151.907393371 /loss/FRt4/assignment_loss‚3.112597829621771 0loss/SRqqt4/assignment_loss‚1.19368155712777 1loss/SRqqt1/detection_loss‚1.0666978555964306 _step‚780ðIÈ7^Ó#Ø 0loss/FRt1/assignment_loss‚1.5751725914685621 /loss/FRt3/detection_loss‚2.6388706355481535 %loss/total_loss‚40.49463730163612 /loss/FBt1/assignment_loss‚0.141345983504921 .loss/FBt2/detection_loss‚2.791399821639061 % _timestamp‚1.7884354672027893e+09 /loss/FRt2/assignment_loss‚2.215552420150943 2loss/SRqqt1/assignment_loss‚0.8040420058387099 /loss/FBt4/detection_loss‚2.5221563246515064 epoch‚7 trainer/global_step‚9499 0loss/SRqqt2/detection_loss‚1.489489731651086 0loss/SRqqt4/detection_loss‚1.995889535292666 1loss/FBt2/assignment_loss‚0.17162520197598496 2loss/FBt3/assignment_loss‚0.027227828948525712 /loss/FRt1/detection_loss‚1.7624319607958845 .loss/FRt2/detection_loss‚2.430091532265268 /loss/FRt4/detection_loss‚2.3638091557051824 1loss/SRqqt3/detection_loss‚2.3432241529226303 /loss/FBt1/detection_loss‚0.5982853825244516 _runtime‚7151.907393371 /loss/FRt4/assignment_loss‚3.112597829621771 0loss/SRqqt4/assignment_loss‚1.19368155712777 1loss/SRqqt1/detection_loss‚1.0666978555964306 _step‚780 2loss/SRqqt2/assignment_loss‚1.3851848445259607 1loss/SRqqt3/assignment_loss‚1.831579381806983 1loss/FBt4/assignment_loss‚0.07959052852432554 0loss/FRt3/assignment_loss‚2.9170303811898104 /loss/FBt3/detection_loss‚3.0376606583595276ŒžƒÄX“Ô#‚(:ˆ Ž¸åÔºÐß gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.797' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mnãr3Õ#‚(:¨ Ž¸åÔË—ó disk.rbd3.out‚0 disk.rbd1.out‚ 3.6796875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚193.9619140625 disk.dm-0.in‚ 1067.51953125 disk.drbd1777.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1067.51953125 disk.drbd1777.out‚0# disk.dm-0.out‚42460.97900390625& disk.nvme0n1.out‚42460.97900390625 disk.rbd1.in‚0" proc.memory.rssMB‚ 2875.0078125% disk./.usageGB‚1100.8209800720215 disk.sda3.out‚ 282.97265625 disk.sda.out‚ 282.97265625 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 950.12109375 cpu‚25.337892769539348( disk.nvme0c0n1.out‚42460.97900390625 network.recv‚ 2.369535e+06) disk./.usagePercent‚73.87701962295702 disk.rbd3.in‚0 disk.dm-1.out‚ 282.97265625! disk.nvme0n1.in‚ 1067.51953125) proc.memory.percent‚17.54765510559082 network.sent‚ 2.2311976e+07 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0$ memory_percent‚94.20092105865479 disk.sda3.in‚0šF5ΓÖ#‚(:ˆ ¸åԦ戎 gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚105.226' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ôbï*×#‚(:Ÿ ¸åÔꘙ’% disk./.usageGB‚1100.8209800720215 disk.dm-0.out‚42560.41015625 disk.rbd2.out‚193.9697265625 disk.drbd1777.in‚0( proc.memory.availableMB‚ 950.33984375 network.recv‚ 2.374181e+06 disk.dm-0.in‚ 1067.51953125# disk.nvme0n1.out‚42560.41015625 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 283.58984375# disk.nvme0c0n1.in‚ 1067.51953125% disk.nvme0c0n1.out‚42560.41015625 cpu‚25.337283088696292 disk.dm-1.out‚ 283.58984375 disk.sda2.in‚0$ memory_percent‚94.19958591461182" proc.memory.rssMB‚ 2875.0078125 network.sent‚ 2.2419502e+07 disk.dm-1.in‚0! disk.nvme0n1.in‚ 1067.51953125 disk.rbd1.out‚ 3.6796875 disk.rbd2.in‚0 disk.sda3.out‚ 283.58984375) proc.memory.percent‚17.54765510559082 proc.cpu.threads‚17) disk./.usagePercent‚73.87701962295702 disk.sda1.out‚0 disk.rbd3.in‚0)9+ “Ø#‚(:ˆ ¬¸åÔßÆã gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.434' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.4868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍÖáƒ.Ù#‚(:£ ¬¸åÔõ„Œ& disk.nvme0n1.out‚42560.67041015625# disk.nvme0c0n1.in‚ 1067.51953125( disk.nvme0c0n1.out‚42560.67041015625 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 950.34375 proc.cpu.threads‚17 network.sent‚ 2.2422904e+07) disk./.usagePercent‚73.87702346307003 disk.rbd3.out‚0 disk.sda.out‚ 284.16015625 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚194.0478515625 network.recv‚ 2.376731e+06 disk.dm-1.out‚ 284.16015625 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0% disk./.usageGB‚1100.8210372924805 disk.dm-0.in‚ 1067.51953125 disk.sda.in‚0 disk.dm-1.in‚0) proc.memory.percent‚17.54765510559082 cpu‚25.33666624886653 disk.rbd1.out‚ 3.6796875 disk.sda3.out‚ 284.16015625 disk.rbd1.in‚0# memory_percent‚94.1995620727539 disk.rbd0.out‚0" proc.memory.rssMB‚ 2875.0078125 disk.rbd3.in‚0# disk.dm-0.out‚42560.67041015625! disk.nvme0n1.in‚ 1067.51953125éX°C2Ú#- _wandbruntime‚7187  _runtime‚7187þÉÇÛ#Á & lr-AdamW/pg2‚2.915060685744675e-05 trainer/global_step‚9549 % _timestamp‚1.7884355023659484e+09 _runtime‚7187.070767791 _step‚781 & lr-AdamW/pg1‚2.915060685744675e-05Ü5ô%ÌÜ#Æ trainer/global_step‚9549 % _timestamp‚1.7884355023659484e+09 _runtime‚7187.070767791 _step‚781 & lr-AdamW/pg1‚2.915060685744675e-05 & lr-AdamW/pg2‚2.915060685744675e-05™ÚY2Ý#- _wandbruntime‚7187  _runtime‚7187ØÈ,©æ'Þ#àO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562420310334464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318729929 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138625842 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚539914695209984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180124125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554430458182144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8381222751232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚554430476185088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180124125 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138625842 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚539914678170624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554430476185088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562420310334464 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7989834149376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚547874447406474 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7989177786880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318749969 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126408022 trainer/global_step‚9549 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562419635969024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80878366 $ _timestamp‚1.788435502366744e+09 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547875118883590 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7960423673606 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45529658 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _runtime‚7187.071681262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138625844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚562419635969024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚554430458182144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8381392666624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7989834149376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚554430476185088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180124125 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138625842 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚539914678170624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554430476185088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562420310334464 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7989834149376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚547874447406474 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7989177786880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318749969 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.frÉ¢Šeed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126408022 trainer/global_step‚9549 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562419635969024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80878366 $ _timestamp‚1.788435502366744e+09 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547875118883590 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7960423673606 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45529658 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _runtime‚7187.071681262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138625844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚562419635969024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚554430458182144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8381392666624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7989834149376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7990653590528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605550049215488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547930167971998 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45534640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7961240108778 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318782465 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚539969582301184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7989997214720 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _step‚783 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180122541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚9549 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80886534 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7989997214720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80886532 % _timestamp‚1.7884355030445044e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126420714 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8382249223680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126421174 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318782465 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8382087710208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚562477482814464 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚562476808435712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚554486811220992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚547930839449322 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7960585670814 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 7187.74959926 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318762399 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138639858 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7990653590528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚562477482814464t€~Üõ&â#ïM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318762399 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180142605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚539969599340544 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180142605 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚554486829223936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚554486811220992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138639858 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7989997214720 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _step‚783 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180122541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚9549 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80886534 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7989997214720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80886532 % _timestamp‚1.7884355030445044e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126420714 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8382249223680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126421174 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318782465 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8382087710208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚562477482814464 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚562476808435712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚554486811220992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚547930839449322 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7960585670814 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 7187.74959926 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318762399 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138639858 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7990653590528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚562477482814464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613932298439168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7990653590528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605550049215488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547930167971998 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45534640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7961240108778 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318782465 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚539969582301184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 320356781 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚557248110124544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884355381628325e+09 _step‚786 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚565278273717248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚565278273717248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45758643 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181030139 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8001259013014 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚608570911420928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430¤æžƒì'ï#æO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 320356781 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚557248110124544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884355381628325e+09 _step‚786 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚565278273717248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚565278273717248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45758643 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181030139 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8001259013014 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚608570911420928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127045872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81286766 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81286764 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8424357937152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139326644 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8030163592704 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181050177 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139326642 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139326642 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181030139 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚557248128127488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8030163592704 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139326644 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320376821 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127045407 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8030819955200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542659884698624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45759106’óÿ„2ð#- _wandbruntime‚7223  _runtime‚7223fØR¼ñ&ñ#ëM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81294934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181048593 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8425213276672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181048593 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81294932 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320389251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542714805868544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚7223.534411504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320409317 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127058606 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320409317 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139340660 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8425043374592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884355388292596e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127058142 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚557304481166336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181068657 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚787 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139340658 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚565336120562688 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8031639396352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8031639396352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚565335446183936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚608632586098688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45763210 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8002075448186 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚550716209839406 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚557304481166336 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139340658 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8030983020544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605Ólf³o ò#ðM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320409317 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139340660 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8425043374592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884355388292596e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127058142 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚557304481166336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181068657 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚787 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139340658 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚565336120562688 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8031639396352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8031639396352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚565335446183936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚608632586098688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45763210 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8002075448186 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚550716209839406 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚557304481166336 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139340658 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8030983020544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81294934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181048593 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8425213276672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181048593 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81294932 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320389251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542714805868544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚7223.534411504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320409317 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127058606“‡xfÎ2ó#- _wandbruntime‚7223  _runtime‚7223[óˆÁWô#Ñ .loss/FRt3/detection_loss‚2.879831506808599 0loss/SRqqt3/detection_loss‚2.499636809934269 /loss/FBt1/detection_loss‚0.6371430626579122 .loss/FBt2/detection_loss‚2.594417800505956 0loss/FRt4/assignment_loss‚2.5584395469403733 -loss/FRt4/detection_loss‚2.07588430537897 0loss/SRqqt2/detection_loss‚2.114231660177833 trainer/global_step‚9599 2loss/SRqqt1/assignment_loss‚0.9323426846973598 2loss/SRqqt4/assignment_loss‚1.3750715982168913 0loss/FBt3/assignment_loss‚0.6706413096528396 /loss/FRt2/detection_loss‚2.6269905770697246 _runtime‚7223.535561781 2loss/SRqqt3/assignment_loss‚1.5448745059018785 0loss/FBt4/assignment_loss‚0.3695608801499475 /loss/FRt1/detection_loss‚1.6882224007751103 %loss/total_loss‚40.71433752053168 .loss/FBt3/detection_loss‚2.443426592482461 % _timestamp‚1.7884355388309345e+09 _step‚788 /loss/FBt4/detection_loss‚1.7662920355796814 epoch‚7 0loss/FRt1/assignment_loss‚1.4612422320577834 0loss/FRt2/assignment_loss‚3.0837308764457703 2loss/SRqqt2/assignment_loss‚1.4319660882465541 /loss/FBt2/assignment_loss‚0.300308924800144 1loss/SRqqt1/detection_loss‚0.9499110163251558 /loss/FRt3/assignment_loss‚3.012166271607081 1loss/FBt1/assignment_loss‚0.06959892444977686 1loss/SRqqt4/detection_loss‚1.6284059096696013Ÿ«\õ#Ö % _timestamp‚1.7884355388309345e+09 _step‚788 /loss/FBt4/detection_loss‚1.7662920355796814 epoch‚7 0loss/FRt1/assignment_loss‚1.4612422320577834 0loss/FRt2/assignment_loss‚3.0837308764457703 2loss/SRqqt2/assignment_loss‚1.4319660882465541 /loss/FBt2/assignment_loss‚0.300308924800144 1loss/SRqqt1/detection_loss‚0.9499110163251558 /loss/FRt3/assignment_loss‚3.012166271607081 1loss/FBt1/assignment_loss‚0.06959892444977686 1loss/SRqqt4/detection_loss‚1.6284059096696013 .loss/FRt3/detection_loss‚2.879831506808599 0loss/SRqqt3/detection_loss‚2.499636809934269 /loss/FBt1/detection_loss‚0.6371430626579122 .loss/FBt2/detection_loss‚2.594417800505956 0loss/FRt4/assignment_loss‚2.5584395469403733 -loss/FRt4/detection_loss‚2.07588430537897 0loss/SRqqt2/detection_loss‚2.114231660177833 trainer/global_step‚9599 2loss/SRqqt1/assignment_loss‚0.9323426846973598 2loss/SRqqt4/assignment_loss‚1.3750715982168913 0loss/FBt3/assignment_loss‚0.6706413096528396 /loss/FRt2/detection_loss‚2.6269905770697246 _runtime‚7223.535561781 2loss/SRqqt3/assignment_loss‚1.5448745059018785 0loss/FBt4/assignment_loss‚0.3695608801499475 /loss/FRt1/detection_loss‚1.6882224007751103 %loss/total_loss‚40.71433752053168 .loss/FBt3/detection_loss‚2.443426592482461”VðèK“ö#‚(:ˆ Ù¸åÔ–¯ÛŽ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.707' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶-‡y-÷#‚(:¢ Ù¸åÔ¥ˆà% disk./.usageGB‚1100.8208503723145 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 286.26953125 disk.drbd1777.in‚0 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚42859.1259765625 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2875.0078125 proc.cpu.threads‚17 network.sent‚ 2.2545831e+07 disk.dm-0.in‚ 1067.51953125# disk.nvme0c0n1.in‚ 1067.51953125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 286.26953125! disk.nvme0n1.in‚ 1067.51953125 disk.rbd2.in‚0 disk.rbd2.out‚195.6533203125$ memory_percent‚94.20459270477295) proc.memory.percent‚17.54765510559082 network.recv‚ 2.3975e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.6796875 cpu‚25.33491801394094) disk./.usagePercent‚73.87701091870086" disk.dm-0.out‚42859.1259765625 disk.sda.in‚0 disk.sda.out‚ 286.26953125 disk.sda1.out‚0% disk.nvme0n1.out‚42859.1259765625( proc.memory.availableMB‚ 949.51953125 =èbø#‚(:’ è¸åÔÀä gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚96.022' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.204400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äL¯.ù#‚(:£ è¸åÔŠ¬ network.recv‚ 2.403659e+06# disk.nvme0c0n1.in‚ 1067.51953125" proc.memory.rssMB‚ 2875.0078125 network.sent‚ 2.265416e+07 disk.sda2.in‚0) proc.memory.percent‚17.54765510559082" disk.dm-0.out‚42958.1494140625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd2.in‚0$ memory_percent‚94.20270919799805& proc.memory.availableMB‚ 949.828125% disk./.usageGB‚1100.8209075927734 disk.sda3.in‚0 disk.sda3.out‚ 286.85546875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 286.85546875' disk.nvme0c0n1.out‚42958.1494140625! disk.rbd2.out‚195.75537109375 cpu‚25.334334018495777 proc.cpu.threads‚17) disk./.usagePercent‚73.87701475881386 disk.dm-0.in‚ 1067.51953125 disk.sda.out‚ 286.85546875% disk.nvme0n1.out‚42958.1494140625 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875 disk.rbd3.out‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 1067.51953125 disk.sda2.out‚0 disk.rbd0.out‚0åhßc2ú#- _wandbruntime‚7258  _runtime‚7258ùʤÒÈû# _runtime‚7258.602010455 _step‚789 ' lr-AdamW/pg1‚2.6995413755433095e-05 ' lr-AdamW/pg2‚2.6995413755433095e-05 trainer/global_step‚9649 $ _timestamp‚1.788435573897263e+09ª7ëÍü#Ç ' lr-AdamW/pg2‚2.6995413755433095e-05 trainer/global_step‚9649 $ _timestamp‚1.788435573897263e+09 _runtime‚7258.602010455 _step‚789 ' lr-AdamW/pg1‚2.6995413755433095e-05•©µy2ý#- _wandbruntime‚7258  _runtime‚7258…,à˜ç'þ#áO BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127683789 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 % _timestamp‚1.7884355738980005e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8071805761024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚611653472244224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚790 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚568136911465472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8467395098112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8071805761024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8071149398528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45988623 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322003673 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚545405091226624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8041439914666 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8467225182720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚560065780069888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚568137585830912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181976229 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127683324 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚568137585830912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45988160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181956191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620120673486848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321983633 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8042094352422üìàÇ$ÿ#æO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8071805761024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8071149398528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45988623 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322003673 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚545405091226624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8041439914666 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8467225182720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚560065780069888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚568137585830912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181976229 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127683324 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚568137585830912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45988160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181956191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620120673486848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321983633 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8042094352422 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127683789 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 % _timestamp‚1.7884355738980005e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8071805761024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚2h†*Ñ%4401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚611653472244224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚790 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚568136911465472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8467395098112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392–©ÉQØ2€$- _wandbruntime‚7259  _runtime‚7259‘ @ñ&$ëM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140041458 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚791 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127696051 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81703332 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚560122133108736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181994709 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140041458 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚553502251706814 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚545460012396544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚568194758310912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 322016103 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8071968826368 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚611715122981888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚545459995357184 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8072625202176 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚9649 _runtime‚7259.277058263 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140041460 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884355745719886e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚568194758310912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚568194083932160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322036169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45992719 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚620183367685120 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚553502923184138 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚560122115105792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚560122115105792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8042256349630 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322016103 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181974645 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0ùäQÑö&‚$ðM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181994709 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚620183171745792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 322036169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8071968826368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚560122133108736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127696503 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8468072704000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8072625202176 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚9649 _runtime‚7259.277058263 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140041460 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884355745719886e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚568194758310912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚568194083932160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322036169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45992719 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚620183367685120 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚553502923184138 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚560122115105792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚560122115105792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8042256349630 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322016103 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181974645 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140041458 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚791 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127696051 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81703332 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚560122133108736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181994709 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140041458 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚553502251706814 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚545460012396544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚568194758310912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 322016103 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8071968826368 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚611715122981888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚545459995357184 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 Fžˆ$‚(:“ †¹åÔúšö gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.523' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.104599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Éj51‰$‚(:¦ †¹åÔ¶Þ…‘ disk.rbd1.in‚0( disk.nvme0c0n1.out‚43060.90380859375 disk.rbd2.in‚0 cpu‚25.33312561729275 proc.cpu.threads‚17 disk.rbd1.out‚ 3.6796875! disk.rbd2.out‚197.32958984375 disk.rbd3.in‚0# disk.dm-0.out‚43060.90380859375 disk.sda.in‚0# disk.nvme0c0n1.in‚ 1067.51953125$ memory_percent‚94.20411586761475& disk.nvme0n1.out‚43060.90380859375 disk.rbd0.in‚0( proc.memory.availableMB‚ 949.59765625 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.54765510559082 network.recv‚ 2.421489e+06) disk./.usagePercent‚73.87703754348442 disk.sda2.out‚0" proc.memory.rssMB‚ 2875.0078125 network.sent‚ 2.2772155e+07# disk./.usageGB‚1100.82124710083 disk.sda3.in‚0 disk.sda3.out‚ 288.27734375 disk.rbd3.out‚0 disk.dm-0.in‚ 1067.51953125 disk.sda.out‚ 288.27734375 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 288.27734375! disk.nvme0n1.in‚ 1067.51953125ƒ@îŠ$‚(:„ •¹åÔ±°ì gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚86.96' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.392 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢¡þ0‹$‚(:¥ •¹åÔ𚧑! disk.rbd2.out‚197.34521484375 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1067.51953125 disk.sda3.in‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2882.46484375$ disk./.usageGB‚1100.821189880371 disk.sda1.in‚0 disk.dm-1.out‚ 288.86328125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 1067.51953125% disk.nvme0n1.out‚43160.2353515625' disk.nvme0c0n1.out‚43160.2353515625 disk.drbd1777.in‚0( proc.memory.availableMB‚ 942.00390625 disk.sda2.in‚0 cpu‚25.342760552194022( disk./.usagePercent‚73.8770337033714 disk.sda.in‚0 disk.sda2.out‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 1067.51953125 disk.rbd0.out‚0" disk.dm-0.out‚43160.2353515625$ memory_percent‚94.25046443939209) proc.memory.percent‚17.59316921234131 network.sent‚ 2.2778554e+07 disk.sda.out‚ 288.86328125 disk.sda3.out‚ 288.86328125 disk.rbd3.out‚0 network.recv‚ 2.426196e+06 disk.dm-1.in‚0 disk.rbd1.out‚ 3.6796875îRô“Œ$‚(:ˆ ¤¹åÔ‹™è gpu.0.gpu‚48.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚157.829' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.5658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸^Ìê0$‚(:¥ ¤¹åÔ–È# disk.dm-0.out‚43259.36279296875 disk.dm-1.in‚0& disk.nvme0n1.out‚43259.36279296875# disk.nvme0c0n1.in‚ 1067.51953125( proc.memory.availableMB‚ 951.31640625# disk./.usageGB‚1100.82124710083 disk.rbd0.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.sda1.out‚0) disk./.usagePercent‚73.87703754348442 disk.rbd2.out‚ 197.431640625 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0 cpu‚25.351923135512955 disk.sda3.out‚ 289.37890625 network.sent‚ 2.2784232e+07 network.recv‚ 2.430243e+06 disk.rbd0.out‚0$ memory_percent‚94.19362545013428 disk.dm-0.in‚ 1067.51953125! disk.nvme0n1.in‚ 1067.51953125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875* proc.memory.percent‚17.537975311279297 disk.dm-1.out‚ 289.37890625 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda.out‚ 289.37890625 disk.sda1.in‚0( disk.nvme0c0n1.out‚43259.36279296875! proc.memory.rssMB‚ 2873.421875¬ è‘Ž$‚(:† ³¹åÔòÇí gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚86.759' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.3518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d¡•\0$‚(:¥ ³¹åÔúÅ… network.sent‚ 2.2790802e+07 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 289.98828125 disk.drbd1777.in‚0 network.recv‚ 2.435215e+06( disk./.usagePercent‚73.8770337033714 disk.sda.out‚ 289.98828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 289.98828125 disk.sda2.in‚0$ disk./.usageGB‚1100.821189880371 disk.dm-1.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0! proc.memory.rssMB‚ 2873.421875 proc.cpu.threads‚17 disk.sda.in‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1067.51953125 disk.drbd1777.out‚0 cpu‚25.361993284583534 disk.dm-0.in‚ 1067.51953125# disk.dm-0.out‚43260.60986328125( proc.memory.availableMB‚ 951.53515625 disk.sda3.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0$ memory_percent‚94.19229030609131! disk.nvme0n1.in‚ 1067.51953125( disk.nvme0c0n1.out‚43260.60986328125 disk.rbd2.out‚ 197.447265625* proc.memory.percent‚17.537975311279297& disk.nvme0n1.out‚43260.60986328125 disk.rbd1.out‚ 3.6796875LIÕ2$- _wandbruntime‚7322  _runtime‚7322Xv<‘$ÇT RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 140879304 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8108069332480 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚82201724 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚623432414363648 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚556475498291686 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚571275133537280 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚548397280444416 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚571274622704128 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 143478272 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8078708761662 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚157 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 _QDeviceStatsMonitor.on_validation_bÌ…uO$atch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 182742630 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8108562162688 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚556476006245438 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 323637913 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚563166553371648 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 119538176 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8108069332480 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 140879306 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚82201726 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _runtime‚7322.075743364 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚46317219 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8108562162688 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚548397297483776 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 323621934 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚614927350152192 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 182742630 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 323637913 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚563166553371648 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚563166571374592 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚9671 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 182758607 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8078217847270 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 140879306 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 128518943 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 8505207689728 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 128519100 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚155 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚46317374 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚623432557841920 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚571274622704128 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 323621934@!R*’$ÌT MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 323621934 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚614927350152192 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 182742630 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 323637913 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚563166553371648 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚563166571374592 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚9671 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 182758607 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8078217847270 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 140879306 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 128518943 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 8505207689728 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 128519100 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚155 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚46317374 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚623432557841920 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚571274622704128 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 323621934 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 140879304 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8108069332480 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚82201724 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚623432414363648 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚556475498291686 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚571275133537280 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚548397280444416 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚571274622704128 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 143478272 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8078708761662 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚157 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 182742630 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8108562162688 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚556476006245438 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 323637913 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚563166553371648 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 119538176 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8108069332480 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 140879306 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚82201726 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _runtime‚7322.075743364 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚46317219 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8108562162688 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚548397297483776 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0™8>›Ÿ2“$- _wandbruntime‚7322  _runtime‚7322×WDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884356382688813e+09 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚563203383400448 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 182749932 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚556511663614334 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 140890112 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 8108927289344 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8108927289344 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚548432591243264 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚571311817859584 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚157 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 119538176 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 128530118 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚794 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 182765909 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8505453278208 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 _runtime‚7322.974019982 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚82208228 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚614965201945600 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚556511155660582 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚623470655223808 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚614965225885696 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 182765909 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚563203383400448 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 323656021 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8505572816384 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 323656021 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚571311817859584 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚563203401403392 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 323640042 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚82208230 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚155 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8108434459136 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 143478272 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 182749932 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 140890110 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚571312328692736 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚623470798702080 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚548432574203904 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚571311817859584 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚563203401403392 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 323640042 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚82208230 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚155 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8108434459136 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 143478272 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 182749932 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 140890110 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚571312328692736 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚623470798702080 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚548432574203904 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884356382688813e+09 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚563203383400448 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 182749932 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚556511663614334 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 140890112 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 8108927289344 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8108927289344 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚548432591243264 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚571311817859584 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚157 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 119538176 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 128530118 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚794 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 182765909 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8505453278208 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152šÁ~¬¥’–$‚(:‡ ¹åÔ§‡îŽ gpu.0.gpu‚59.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.476' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OB‡"—$‚(:— ¹åÔ•‘”! proc.memory.rssMB‚ 2873.421875 proc.cpu.threads‚17% disk./.usageGB‚1100.8211936950684 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda.out‚290.4375 disk.sda1.in‚0% disk.nvme0n1.out‚43359.8955078125 disk.rbd2.in‚0* proc.memory.percent‚17.537975311279297( proc.memory.availableMB‚ 953.36328125 network.recv‚ 2.439038e+06) disk./.usagePercent‚73.87703395937893$ memory_percent‚94.18113231658936 disk.sda3.out‚290.4375 disk.dm-0.in‚ 1067.51953125 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 1067.51953125 disk.rbd0.in‚0 disk.rbd1.out‚ 3.6796875 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd2.out‚ 197.509765625 cpu‚25.371138009409282 network.sent‚ 2.279591e+07 disk.drbd1777.out‚0" disk.dm-0.out‚43359.8955078125 disk.sda.in‚0 disk.dm-1.out‚290.4375! disk.nvme0n1.in‚ 1067.51953125 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚43359.8955078125 disk.rbd0.out‚0½¥2˜$- _wandbruntime‚7335  _runtime‚7335A÷w!ŠX™$À¦ 7validation_average_jet_accuracy‚0.44122520089149475 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9924209117889404 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 particle/accuracy_7_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5249956250190735 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16797681152820587 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16363541781902313 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5091627836227417 particle/accuracy_4_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5419512987136841 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.39534884691238403 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26057952642440796 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891391396522522 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16239464282989502 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17491081357002258 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7679556012153625 ,jet/accuracy_4_of_5‚0.012634407728910446 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48492172360420227 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8745254278182983 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4418826997280121 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6998578906059265 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022868139669299126 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985267162322998 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5133925080299377 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4801587462425232 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 particle/accuracy_6_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.19230769574642181 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6381765007972717 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9858636856079102 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4898677468299866 particle/accuracy_1_of_3‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5196706056594849 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2261904776096344 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9594109058380127 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28447669744491577 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.10606061667203903 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5117781162261963 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.561904788017273 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884356507026496e+09 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9912156462669373 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41294050216674805 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_6_of_6‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5100979804992676 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37812498211860657 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2183235138654709 *jet/accuracy_1_of_5‚0.9581349492073059 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19902309775352478 +jet/accuracy_3_of_5‚0.09361239522695541 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322917461395264 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.47755441069602966 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0013227513991296291 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35890597105026245 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15180742740631104 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47496795654296875 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08062324672937393 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6578947305679321 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1857294887304306 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 particle/accuracy_5_of_6‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41463881731033325 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9927951097488403 ,jet/accuracy_3_of_3‚0.023659182712435722 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333134651184 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03106279857456684 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5217092037200928 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 _step‚795 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9848790168762207 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20603448152542114 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5204091668128967 particle/accuracy_1_of_4‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3819444477558136 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.49411463737487793 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5022351145744324 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5739394426345825 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9934163689613342 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ,particle/sensitivity‚0.30898410081863403 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.2155555784702301 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9501469731330872 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333134651184 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997860848903656 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 _runtime‚7335.409417006 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17013432085514069 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5017998218536377 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.20800825953483582 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04374999925494194 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5363714694976807 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,jet/accuracy_4_of_4‚0.008048078045248985 particle/accuracy_4_of_6‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14750957489013672 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0516321025788784 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35609254240989685 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6070468425750732 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14842215180397034 'particle/accuracy‚0.850027859210968 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9900197982788086 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2087155431509018 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.021579744294285774 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.43281927704811096 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17202316224575043 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.277777761220932 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 trainer/global_step‚9671 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27156662940979004 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9736239314079285 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10048021376132965 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3689153790473938 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_7‚0.5 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5321668982505798 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.016439909115433693 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5186781883239746 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878787517547607 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30302420258522034 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01948499120771885 particle/accuracy_1_of_2‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.40936383605003357 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05924449861049652 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.25441431999206543 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17477628588676453 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.554500937461853 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24747473001480103 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3194129168987274 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.18421052396297455 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 particle/accuracy_3_of_5‚1 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333432674408 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3891277015209198 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39784231781959534 particle/accuracy_2_of_7‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2690669894218445 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23983582854270935 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1645934134721756 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4479166865348816 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6170138716697693 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47945404052734375 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2291577309370041 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4364035129547119 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5474528074264526 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02604166604578495 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20448867976665497 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18112848699092865 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5131062865257263 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 +jet/accuracy_3_of_4‚0.07316923141479492 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4116858243942261 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 jet/accuracy_1_of_6‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34107139706611633 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333432674408 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11956153810024261 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5169752836227417 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.07#û½:431547611951828 *jet/accuracy_1_of_4‚0.9251876473426819 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2764681875705719 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2567567527294159 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2302263081073761 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_5_of_6‚0.01923076994717121 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.39506176114082336 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5304925441741943 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25127938389778137 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5447390675544739 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 particle/accuracy_1_of_7‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1881648451089859 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15925614535808563 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4143097698688507 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16277329623699188 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.487607479095459 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9892479181289673 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4926566779613495 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5546347498893738 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1806119829416275 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5506687164306641 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19996464252471924 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.48734167218208313 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3636363744735718 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.43549028038978577 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18260052800178528 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_6‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8095677495002747 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5038472414016724 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5555555820465088 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_7‚1 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 jet/accuracy_5_of_5‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4583878517150879 *"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17816898226737976 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9912607073783875 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17816898226737976 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03739682212471962 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23105379939079285 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9876857995986938 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5100979804992676 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 +jet/accuracy_2_of_6‚0.49358975887298584 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07452961057424545 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2678530216217041 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 jet/accuracy_1_of_7‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.488402783870697 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10611728578805923 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004828328732401133 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5498185157775879 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 epoch‚7 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.40630632638931274 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.34166666865348816 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5100591778755188 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9208037853240967 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5167545080184937 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.30762651562690735 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4292159676551819 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.48859071731567383 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983847737312317 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2086666375398636 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.202031210064888 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5512129664421082 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5995111465454102 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4472767412662506 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 +jet/accuracy_2_of_4‚0.39418432116508484 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.551177442073822 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02480551227927208 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.08579638600349426 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24922841787338257 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5726600885391235 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19883041083812714 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18386846780776978 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008935296908020973 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_3_of_3‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5897017121315002 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0851370319724083 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4171539843082428 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18436004221439362 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6043498516082764 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 +"Purity/0FRt0SRqqt3FBt/event_purity‚0.25 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15226337313652039 particle/accuracy_1_of_5‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5302186012268066 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22120517492294312 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25297579169273376 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4248909056186676 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 *loss/val_total_loss‚41.792178427400636 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5661622285842896 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 'particle/f_score‚0.3913467228412628 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31111109256744385 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1808510571718216 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9892466068267822 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49420589208602905 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3704102039337158 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 particle/accuracy_5_of_5‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.22962966561317444 ,jet/accuracy_4_of_6‚0.057692307978868484 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.41927871108055115 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 particle/accuracy_1_of_1‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013513168320059776 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.34375 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1021614819765091 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9837201237678528 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2580459415912628 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0´ÌuÄ5Eš$Ŧ 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4116858243942261 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 jet/accuracy_1_of_6‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34107139706611633 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333432674408 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11956153810024261 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5169752836227417 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 *jet/accuracy_1_of_4‚0.9251876473426819 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2764681875705719 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2567567527294159 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2302263081073761 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_5_of_6‚0.01923076994717121 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.39506176114082336 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5304925441741943 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25127938389778137 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5447390675544739 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 particle/accuracy_1_of_7‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1881648451089859 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15925614535808563 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4143097698688507 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16277329623699188 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.487607479095459 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9892479181289673 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4926566779613495 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5546347498893738 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1806119829416275 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5506687164306641 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19996464252471924 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.48734167218208313 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3636363744735718 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.43549028038978577 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18260052800178528 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_6‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8095677495002747 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5038472414016724 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5555555820465088 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_7‚1 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 jet/accuracy_5_of_5‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4583878517150879 *"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17816898226737976 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9912607073783875 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17816898226737976 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03739682212471962 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23105379939079285 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9876857995986938 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5100979804992676 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 +jet/accuracy_2_of_6‚0.49358975887298584 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07452961057424545 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2678530216217041 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 jet/accuracy_1_of_7‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.488402783870697 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10611728578805923 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004828328732401133 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5498185157775879 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 epoch‚7 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.40630632638931274 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.34166666865348816 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5100591778755188 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9208037853240967 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5167545080184937 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.30762651562690735 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4292159676551819 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.48859071731567383 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983847737312317 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2086666375398636 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.202031210064888 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5512129664421082 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5995111465454102 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4472767412662506 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 +jet/accuracy_2_of_4‚0.39418432116508484 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.551177442073822 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02480551227927208 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.08579638600349426 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24922841787338257 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5726600885391235 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19883041083812714 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18386846780776978 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008935296908020973 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_3_of_3‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5897017121315002 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0851370319724083 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4171539843082428 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18436004221439362 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6043498516082764 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 +"Purity/0FRt0SRqqt3FBt/event_purity‚0.25 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15226337313652039 particle/accuracy_1_of_5‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5302186012268066 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22120517492294312 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25297579169273376 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4248909056186676 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 *loss/val_total_loss‚41.792178427400636 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5661622285842896 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 'particle/f_score‚0.3913467228412628 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31111109256744385 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1808510571718216 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9892466068267822 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49420589208602905 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3704102039337158 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 particle/accuracy_5_of_5‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.22962966561317444 ,jet/accuracy_4_of_6‚0.057692307978868484 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.41927871108055115 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 particle/accuracy_1_of_1‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013513168320059776 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.34375 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1021614819765091 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9837201237678528 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2580459415912628 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.44122520089149475 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9924209117889404 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 particle/accuracy_7_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5249956250190735 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16797681152820587 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16363541781902313 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5091627836227417 particle/accuracy_4_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqÎbKNqt*FBt/FBt_purity‚0.9833333492279053 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5419512987136841 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.39534884691238403 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26057952642440796 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891391396522522 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16239464282989502 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17491081357002258 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7679556012153625 ,jet/accuracy_4_of_5‚0.012634407728910446 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48492172360420227 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8745254278182983 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4418826997280121 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6998578906059265 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022868139669299126 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985267162322998 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5133925080299377 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4801587462425232 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 particle/accuracy_6_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.19230769574642181 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6381765007972717 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9858636856079102 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4898677468299866 particle/accuracy_1_of_3‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5196706056594849 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2261904776096344 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9594109058380127 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28447669744491577 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.10606061667203903 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5117781162261963 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.561904788017273 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884356507026496e+09 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9912156462669373 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41294050216674805 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_6_of_6‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5100979804992676 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37812498211860657 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2183235138654709 *jet/accuracy_1_of_5‚0.9581349492073059 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19902309775352478 +jet/accuracy_3_of_5‚0.09361239522695541 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322917461395264 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.47755441069602966 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0013227513991296291 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35890597105026245 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15180742740631104 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47496795654296875 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08062324672937393 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6578947305679321 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1857294887304306 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 particle/accuracy_5_of_6‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41463881731033325 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9927951097488403 ,jet/accuracy_3_of_3‚0.023659182712435722 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333134651184 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03106279857456684 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5217092037200928 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 _step‚795 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9848790168762207 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20603448152542114 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5204091668128967 particle/accuracy_1_of_4‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3819444477558136 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.49411463737487793 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5022351145744324 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5739394426345825 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9934163689613342 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ,particle/sensitivity‚0.30898410081863403 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.2155555784702301 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9501469731330872 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333134651184 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997860848903656 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 _runtime‚7335.409417006 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17013432085514069 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5017998218536377 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.20800825953483582 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04374999925494194 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5363714694976807 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,jet/accuracy_4_of_4‚0.008048078045248985 particle/accuracy_4_of_6‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14750957489013672 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0516321025788784 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35609254240989685 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6070468425750732 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14842215180397034 'particle/accuracy‚0.850027859210968 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9900197982788086 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2087155431509018 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.021579744294285774 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.43281927704811096 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17202316224575043 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.277777761220932 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 trainer/global_step‚9671 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27156662940979004 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9736239314079285 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10048021376132965 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3689153790473938 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_7‚0.5 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5321668982505798 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.016439909115433693 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5186781883239746 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878787517547607 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30302420258522034 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01948499120771885 particle/accuracy_1_of_2‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.40936383605003357 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05924449861049652 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.25441431999206543 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17477628588676453 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.554500937461853 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24747473001480103 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3194129168987274 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.18421052396297455 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 particle/accuracy_3_of_5‚1 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333432674408 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3891277015209198 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39784231781959534 particle/accuracy_2_of_7‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2690669894218445 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23983582854270935 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1645934134721756 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4479166865348816 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6170138716697693 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47945404052734375 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2291577309370041 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4364035129547119 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5474528074264526 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02604166604578495 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20448867976665497 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18112848699092865 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5131062865257263 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 +jet/accuracy_3_of_4‚0.07316923141479492 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0›΂4ì‘›$‚(:† ѹåÔ•Ž gpu.0.gpu‚50.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r¹'œ$‚(:œ ѹåÔ›æª disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 291.02734375$ disk.nvme0n1.out‚43932.865234375 disk.rbd1.out‚3.765625$ memory_percent‚91.29562377929688$ proc.memory.availableMB‚1426.125& disk.nvme0c0n1.out‚43932.865234375 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 291.02734375* proc.memory.percent‚17.537975311279297 disk.rbd2.out‚ 197.525390625 disk.drbd1777.out‚0! proc.memory.rssMB‚ 2873.421875 cpu‚25.370043020262298 disk.sda.out‚ 291.02734375 network.sent‚ 2.2904736e+07) disk./.usagePercent‚73.87718269975633 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 network.recv‚ 2.445972e+06% disk./.usageGB‚1100.8234100341797 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚43932.865234375 disk.sda1.out‚0 disk.dm-1.in‚0! disk.nvme0n1.in‚ 1067.51953125 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1067.51953125 disk.dm-0.in‚ 1067.51953125qä(:2$- _wandbruntime‚7358  _runtime‚7358¤‘ù¹Çž$Á _runtime‚7358.949958117 _step‚796 & lr-AdamW/pg1‚2.594468802953354e-05 & lr-AdamW/pg2‚2.594468802953354e-05 trainer/global_step‚9699 % _timestamp‚1.7884356742451952e+09~ëf@ÌŸ$Æ _step‚796 & lr-AdamW/pg1‚2.594468802953354e-05 & lr-AdamW/pg2‚2.594468802953354e-05 trainer/global_step‚9699 % _timestamp‚1.7884356742451952e+09 _runtime‚7358.949958117œ5rÕ62 $- _wandbruntime‚7358  _runtime‚7358dHLðç'¡$áO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚617122180338176 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573338860998144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884356742459648e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8135811536896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚565203705823744 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141409022 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚797 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚625655842614784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚550373804602368 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129015816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7358.950918596 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚565203723826688 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183344206 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚617122156398080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚558479675833640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚550373821641728 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8105854191912 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129016256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 324773266 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8533662276608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573339535363584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚565203723826688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8135155174400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573339535363584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46502500 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8533496555520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 324753226 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8135811536896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8135155174400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8105199754156 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141409022 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚625655652953600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141409020 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573338860998144 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82513316 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚558479004356524 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183364244 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183344206 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573338860998144 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82513316 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚558479004356524 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183364244 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183344206 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚625655842614784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚550373804602368 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129015816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7358.950918596 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚565203723826688 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183344206 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚617122156398080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚558479675833640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚550373821641728 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8105854191912 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129016256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 324773266 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8533662276608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573339535363584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚565203723826688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8135155174400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573339535363584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46502500 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8533496555520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 324753226 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8135811536896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0‘,n2£$- _wandbruntime‚7359  _runtime‚7359ikÅTñ&¤$ëM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573396707843584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617183831075840 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 324785696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183362660 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573396033464832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 324785696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚625718348342784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8534351559168 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚565260058862592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129028425 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573396707843584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8135974602240 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚550428725772288 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82521486 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324805762 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183362660 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46507374 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8106670627084 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚798 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183382724 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚558534724922048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8135974602240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚9699 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82521484 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141423038 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141423036 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚565260058862592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183382724 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚558535396399372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324805762 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844rÍp²ö&¥$ðM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚558535396399372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324805762 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573396707843584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617183831075840 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 324785696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183362660 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573396033464832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 324785696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚625718348342784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8534351559168 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚565260058862592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129028425 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573396707843584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8135974602240 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚550428725772288 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82521486 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324805762 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183362660 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46507374 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8106670627084 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚798 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183382724 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚558534724922048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8135974602240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚9699 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82521484 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141423038 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141423036 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚565260058862592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183382724 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928ž —‡±2¦$- _wandbruntime‚7359  _runtime‚7359M[—ñZ§$Ô 1loss/FBt2/assignment_loss‚0.46143417215452326 .loss/FRt2/detection_loss‚2.299720136956735 /loss/FRt3/detection_loss‚2.9559945444266003 .loss/FRt3/assignment_loss‚2.76025128364563 /loss/FRt4/detection_loss‚2.2444102941524413 .loss/FBt3/detection_loss‚3.032579803466797 _runtime‚7359.637089557 2loss/SRqqt2/assignment_loss‚1.4473092331779454 1loss/FBt3/assignment_loss‚0.23599136960692704 1loss/FBt4/assignment_loss‚0.19870376419275998 1loss/SRqqt2/detection_loss‚1.8698827388151638 .loss/FBt2/detection_loss‚2.009221374988556 1loss/SRqqt4/detection_loss‚1.8364131876826286 2loss/SRqqt4/assignment_loss‚1.4340727269276976 .loss/FRt1/detection_loss‚1.507761733494699 .loss/FBt1/detection_loss‚0.573770353656911 _step‚799 2loss/SRqqt1/assignment_loss‚0.9608987427316606 1loss/SRqqt3/assignment_loss‚1.682857770472765 .loss/FBt4/detection_loss‚2.066186398267746 epoch‚8 % _timestamp‚1.7884356749321783e+09 1loss/SRqqt1/detection_loss‚1.1631082798043886 1loss/SRqqt3/detection_loss‚2.4453939229249952 %loss/total_loss‚39.39626750818056 0loss/FRt1/assignment_loss‚1.1490291380882263 0loss/FRt2/assignment_loss‚2.5122172046791422 0loss/FRt4/assignment_loss‚2.4443908504077365 1loss/FBt1/assignment_loss‚0.10466848345788093 trainer/global_step‚9699_í¹_¨$Ù 2loss/SRqqt2/assignment_loss‚1.4473092331779454 1loss/FBt3/assignment_loss‚0.23599136960692704 1loss/FBt4/assignment_loss‚0.19870376419275998 1loss/SRqqt2/detection_loss‚1.8698827388151638 .loss/FBt2/detection_loss‚2.009221374988556 1loss/SRqqt4/detection_loss‚1.8364131876826286 2loss/SRqqt4/assignment_loss‚1.4340727269276976 .loss/FRt1/detection_loss‚1.507761733494699 .loss/FBt1/detection_loss‚0.573770353656911 _step‚799 2loss/SRqqt1/assignment_loss‚0.9608987427316606 1loss/SRqqt3/assignment_loss‚1.682857770472765 .loss/FBt4/detection_loss‚2.066186398267746 epoch‚8 % _timestamp‚1.7884356749321783e+09 1loss/SRqqt1/detection_loss‚1.1631082798043886 1loss/SRqqt3/detection_loss‚2.4453939229249952 %loss/total_loss‚39.39626750818056 0loss/FRt1/assignment_loss‚1.1490291380882263 0loss/FRt2/assignment_loss‚2.5122172046791422 0loss/FRt4/assignment_loss‚2.4443908504077365 1loss/FBt1/assignment_loss‚0.10466848345788093 trainer/global_step‚9699 1loss/FBt2/assignment_loss‚0.46143417215452326 .loss/FRt2/detection_loss‚2.299720136956735 /loss/FRt3/detection_loss‚2.9559945444266003 .loss/FRt3/assignment_loss‚2.76025128364563 /loss/FRt4/detection_loss‚2.2444102941524413 .loss/FBt3/detection_loss‚3.032579803466797 _runtime‚7359.637089557Ÿþ%çRì©$‚(:ˆ à¹åÔµ†Ù gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWas'x§tts‚500.0 gpu.0.powerPercent‚31.3156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±úÏ)ª$‚(:ž à¹åÔš€ó) disk./.usagePercent‚73.87707415256182 disk.dm-1.out‚ 291.359375 network.sent‚ 2.3031639e+07 network.recv‚ 2.458873e+06 disk.sda.out‚ 291.359375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 199.099609375 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0' proc.memory.availableMB‚ 1426.109375 proc.cpu.threads‚17 disk.rbd1.out‚3.828125 disk.sda3.out‚ 291.359375 disk.rbd3.out‚0# disk.dm-0.out‚44056.09033203125 disk.sda.in‚0! disk.nvme0n1.in‚ 1067.51953125& disk.nvme0n1.out‚44056.09033203125( disk.nvme0c0n1.out‚44056.09033203125 disk.sda3.in‚0! proc.memory.rssMB‚ 2873.421875 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1067.51953125 disk.drbd1777.in‚0 disk.dm-0.in‚ 1067.51953125 cpu‚25.369396625086683 disk.sda1.out‚0* proc.memory.percent‚17.537975311279297$ disk./.usageGB‚1100.821792602539 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0$ memory_percent‚91.29571914672852©q:M’«$‚(:‡ ï¹åÔ˜ÆÞ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚90.143' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0286 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åš™a'¬$‚(:œ ï¹åÔ¼”Ê) disk./.usagePercent‚73.87707415256182 disk.rbd3.out‚0 disk.sda.out‚ 292.08984375 disk.rbd1.out‚3.828125 disk.dm-0.out‚44604.33984375 disk.sda1.in‚0# disk.nvme0n1.out‚44604.33984375 disk.rbd2.in‚0! proc.memory.rssMB‚ 2873.421875 network.recv‚ 2.473185e+06 disk.sda.in‚0 disk.dm-1.in‚0% disk.nvme0c0n1.out‚44604.33984375 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 network.sent‚ 2.3144465e+07$ disk./.usageGB‚1100.821792602539 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1425.7265625 cpu‚25.368778626018585 disk.rbd2.out‚ 199.115234375 disk.sda3.out‚ 292.08984375 disk.dm-0.in‚ 1067.51953125! disk.nvme0n1.in‚ 1067.51953125$ memory_percent‚91.29805564880371 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1067.51953125 disk.sda3.in‚0* proc.memory.percent‚17.537975311279297 proc.cpu.threads‚17 disk.dm-1.out‚ 292.08984375 disk.rbd0.out‚0¶µ§ 2­$- _wandbruntime‚7394  _runtime‚7394ôë eÉ®$à ' lr-AdamW/pg1‚2.4912169396160058e-05 ' lr-AdamW/pg2‚2.4912169396160058e-05 trainer/global_step‚9749 % _timestamp‚1.7884357096341684e+09 _runtime‚7394.338908877 _step‚800á±1%ί$È ' lr-AdamW/pg1‚2.4912169396160058e-05 ' lr-AdamW/pg2‚2.4912169396160058e-05 trainer/global_step‚9749 % _timestamp‚1.7884357096341684e+09 _runtime‚7394.338908877 _step‚800 ITï2°$- _wandbruntime‚7394  _runtime‚7394?§ÄMç'±$áO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚628781431985152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚7394.339809173 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚576197498746368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129650770 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8576546945536 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184290296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46729052 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8176140980224 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142109820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884357096348906e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142109822 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚568021375769088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184290296 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚561265717701048 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142109820 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚576198173111808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184270258 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8146035093564 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8576714763776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚568021357766144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8176797342720 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184270258 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8176140980224 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129650327 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚576197498746368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129650770 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8576546945536 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184290296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46729052 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8176140980224 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142109820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884357096348906e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142109822 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚568021375769088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184290296 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚561265717701048 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142109820 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚576198173111808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184270258 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8146035093564 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8576714763776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚568021357766144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8176797342720 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184270258 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8176140980224 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129650327 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326432614 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82929884 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553173932300288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8147505966492 _runtime‚7395.016209215 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129663020 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46733582 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚576255345591808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142123836 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚568077710804992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8577400304640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚568077728807936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8177616783872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8177616783872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142123838 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚576255345591808 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142123836 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚568077710804992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184308776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_traiÓD‹ž n_batch_end/allocated_bytes.all.freed‚576254671213056 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326432614 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82929884 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553173932300288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8147505966492 _runtime‚7395.016209215 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129663020 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46733582 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚576255345591808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142123836 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚568077710804992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8577400304640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚568077728807936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8177616783872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8177616783872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142123838 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚576255345591808 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142123836 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚568077710804992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184308776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚576254671213056 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 Bæ'Ã$àO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83330116 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚631906831397376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚631906656416256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142810622 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8619577292800 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚805 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚7429.907326689 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚570839009708544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185216348 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130285202 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555864234697728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚570839027711488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8217783148544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8187524870728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚579056810860032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚579056810860032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142810620 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 328026970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚623287230164480 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚555864217658368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚579056136494592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 328006930 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8217126786048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚579056136494592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8Dev“#a«NiceStatsMonitor.on_train_batch_start/active.all.freed‚ 328006930 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8217126786048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚7429.907326689 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚570839009708544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185216348 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130285202 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555864234697728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚570839027711488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8217783148544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8187524870728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚579056810860032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚579056810860032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142810620 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 328026970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚623287230164480 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚555864217658368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944¥ úí_2Å$- _wandbruntime‚7430  _runtime‚7430³>ƒñ&Æ$ëM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46959978 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚7430.570496687 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚579113308961280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚564106808656864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142824638 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚564107480134188 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185234828 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142824638 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130298264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚631969161055744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8620447958528 trainer/global_step‚9799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328059466 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚570895362747392 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚623348880902144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328059466 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8217946213888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8218602589696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83338286 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185214764 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185234828 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8217946213888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8187686867936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83338284 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚579113983340032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 328039400 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142824636 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46959554 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142824636 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8218602589696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555919138828288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚555919121788928 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8620280153600 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130297838 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884357458652198e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚579113983340032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚570895380750336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605v’uyö&Ç$ðM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚579113308961280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚564106808656864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142824638 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚564107480134188 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185234828 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142824638 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130298264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚631969161055744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8620447958528 trainer/global_step‚9799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328059466 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚570895362747392 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚623348880902144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328059466 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8217946213888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8218602589696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83338286 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185214764 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185234828 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8217946213888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8187686867936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83338284 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚579113983340032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 328039400 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142824636 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46959554 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142824636 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8218602589696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555919138828288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚555919121788928 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8620280153600 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130297838 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884357458652198e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚579113983340032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚570895380750336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46959978 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚7430.570496687¦¢¡!2È$- _wandbruntime‚7430  _runtime‚7430{]WÉ$Ñ 1loss/SRqqt1/detection_loss‚0.9294019620865583 1loss/SRqqt2/detection_loss‚1.4874035540435995 1loss/SRqqt4/detection_loss‚1.8128859528473444 /loss/FBt4/detection_loss‚2.0338555466045034 epoch‚8 _runtime‚7430.571812095 .loss/FRt2/detection_loss‚2.671078399187181 1loss/SRqqt2/assignment_loss‚1.203114432037442 1loss/FBt1/assignment_loss‚0.07200125374992526 /loss/FBt3/assignment_loss‚0.276931956410408 /loss/FBt1/detection_loss‚0.5844164713466146 2loss/SRqqt3/assignment_loss‚1.7965602593556527 .loss/FRt3/detection_loss‚2.578053184245762 %loss/total_loss‚38.87667206271142 0loss/FRt2/assignment_loss‚2.9585428935725515 2loss/SRqqt1/assignment_loss‚0.7746266029684598 % _timestamp‚1.7884357458668697e+09 _step‚807 .loss/FRt3/assignment_loss‚2.77391822400846 0loss/FBt2/assignment_loss‚0.5356819998531137 .loss/FBt2/detection_loss‚2.489187145233154 /loss/FBt3/detection_loss‚2.2837715016471014 1loss/SRqqt4/assignment_loss‚1.644987170489467 1loss/FBt4/assignment_loss‚0.22287077265156602 .loss/FRt1/detection_loss‚1.734804568514228 /loss/FRt4/detection_loss‚1.9097084212810504 0loss/SRqqt3/detection_loss‚2.338549112119982 trainer/global_step‚9799 0loss/FRt1/assignment_loss‚0.9277389359474182 0loss/FRt4/assignment_loss‚2.8365817425098827µÝ—Þ\Ê$Ö 0loss/FRt2/assignment_loss‚2.9585428935725515 2loss/SRqqt1/assignment_loss‚0.7746266029684598 % _timestamp‚1.7884357458668697e+09 _step‚807 .loss/FRt3/assignment_loss‚2.77391822400846 0loss/FBt2/assignment_loss‚0.5356819998531137 .loss/FBt2/detection_loss‚2.489187145233154 /loss/FBt3/detection_loss‚2.2837715016471014 1loss/SRqqt4/assignment_loss‚1.644987170489467 1loss/FBt4/assignment_loss‚0.22287077265156602 .loss/FRt1/detection_loss‚1.734804568514228 /loss/FRt4/detection_loss‚1.9097084212810504 0loss/SRqqt3/detection_loss‚2.338549112119982 trainer/global_step‚9799 0loss/FRt1/assignment_loss‚0.9277389359474182 0loss/FRt4/assignment_loss‚2.8365817425098827 1loss/SRqqt1/detection_loss‚0.9294019620865583 1loss/SRqqt2/detection_loss‚1.4874035540435995 1loss/SRqqt4/detection_loss‚1.8128859528473444 /loss/FBt4/detection_loss‚2.0338555466045034 epoch‚8 _runtime‚7430.571812095 .loss/FRt2/detection_loss‚2.671078399187181 1loss/SRqqt2/assignment_loss‚1.203114432037442 1loss/FBt1/assignment_loss‚0.07200125374992526 /loss/FBt3/assignment_loss‚0.276931956410408 /loss/FBt1/detection_loss‚0.5844164713466146 2loss/SRqqt3/assignment_loss‚1.7965602593556527 .loss/FRt3/detection_loss‚2.578053184245762 %loss/total_loss‚38.87667206271142§ÕG“Ë$‚(:ˆ «ºåÔ“©ç gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.729' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.3458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çÎy)Ì$‚(:ž «ºåÔáÓÛ disk.sda.out‚294.4375# disk.nvme0c0n1.in‚ 1067.51953125! proc.memory.rssMB‚ 2873.421875 disk.rbd1.in‚0 disk.rbd0.out‚0! disk.rbd2.out‚200.80322265625 disk.drbd1777.in‚0 disk.sda.in‚0( disk.nvme0c0n1.out‚44902.47705078125 disk.sda2.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0 cpu‚25.366182310421557 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚44902.47705078125 disk.rbd1.out‚ 3.92333984375* proc.memory.percent‚17.537975311279297 network.sent‚ 2.3361753e+07 network.recv‚ 2.515991e+06 disk.dm-0.in‚ 1067.51953125% proc.memory.availableMB‚ 1421.8125% disk./.usageGB‚1100.8218612670898 disk.rbd3.out‚0 disk.dm-1.in‚0! disk.nvme0n1.in‚ 1067.51953125 disk.rbd0.in‚0# disk.dm-0.out‚44902.47705078125 disk.dm-1.out‚294.4375 disk.sda3.out‚294.4375$ memory_percent‚91.32194519042969 proc.cpu.threads‚17) disk./.usagePercent‚73.87707876069743׌â]“Í$‚(:ˆ ººåÔŽÚ‚Ž gpu.0.gpu‚37.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.729' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.3458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îqcÎ$‚(:“ ººåÔ‘Àª‘ disk.rbd0.in‚0' proc.memory.availableMB‚ 1423.578125! disk.rbd2.out‚200.86572265625 disk.dm-0.out‚44902.83984375 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1067.51953125% disk.nvme0c0n1.out‚44902.83984375! proc.memory.rssMB‚ 2873.421875 network.recv‚ 2.521848e+06) disk./.usagePercent‚73.87708029674263 disk.rbd2.in‚0 disk.sda.out‚294.9375 disk.rbd1.out‚ 3.986328125% disk./.usageGB‚1100.8218841552734 disk.sda3.in‚0 disk.dm-1.in‚0# disk.nvme0n1.out‚44902.83984375 disk.rbd1.in‚0 disk.sda2.out‚0 cpu‚25.36551064249362 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 1067.51953125 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0! disk.nvme0n1.in‚ 1067.51953125* proc.memory.percent‚17.537975311279297 network.sent‚ 2.3469673e+07 disk.sda3.out‚294.9375 disk.dm-1.out‚294.9375# memory_percent‚91.3111686706543 disk.sda2.in‚0©˜2Ï$- _wandbruntime‚7465  _runtime‚7465 ¹ÙlÈÐ$ ' lr-AdamW/pg1‚2.2902610269160546e-05 ' lr-AdamW/pg2‚2.2902610269160546e-05 trainer/global_step‚9849 $ _timestamp‚1.788435780800594e+09 _runtime‚7465.505307218 _step‚808÷°&œÍÑ$Ç ' lr-AdamW/pg2‚2.2902610269160546e-05 trainer/global_step‚9849 $ _timestamp‚1.788435780800594e+09 _runtime‚7465.505307218 _step‚808 ' lr-AdamW/pg1‚2.2902610269160546e-05¨,g»ÕÒ$- _wandbruntim"Â]Ce‚7465  _runtime‚7465Eôwç'Ó$áO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83738518 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚581915448608256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8662634981888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47182831 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚7465.506210129 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83738516 trainer/global_step‚9849 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚581914774242816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚635032242599936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚809 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 186142400 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186122362 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186122362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚558609424186368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚573656661650944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚573656679653888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573656661650944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884357808012927e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚581915448608256 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143511420 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8258768954368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47183276 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚581914774242816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130921347 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚626369790987776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130921794 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚566837129958748 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚558609441225728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8227705772380 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143511422 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8258768954368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8662475552256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚573656679653888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143511422 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908¯ë‹jì'Ô$æO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8228360210136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329633782 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329653822 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143511420 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚573656679653888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573656661650944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884357808012927e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚581915448608256 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143511420 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8258768954368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47183276 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚581914774242816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130921347 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚626369790987776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130921794 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚566837129958748 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚558609441225728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8227705772380 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143511422 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8258768954368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8662475552256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚573656679653888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143511422 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83738518 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚581915448608256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8662634981888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47182831 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚7465.506210129 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83738516 trainer/global_step‚9849 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚581914774242816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚635032242599936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚809 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 186142400 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186122362 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186122362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚558609424186368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚573656661650944 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186160880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 _step‚810 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚566893522001596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83746684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚635094935728128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚566892850524272 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚558664345356288 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚573713032692736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8258932019712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚635094739788800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329686318 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8663322003456 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143525436 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83746686 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚573713032692736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130934612 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚9849 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143525436 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚573713014689792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329666252 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130934146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581971946709504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8229176645308 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186140816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚581972621088256 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329686318 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8259588395520 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8259588395520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47187462 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚7466.403303098"Ž^²Å×$ðM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8663494002688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186160880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 _step‚810 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚566893522001596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83746684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚635094935728128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚566892850524272 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚558664345356288 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚573713032692736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8258932019712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚635094739788800 ;4DeviceStatsMonitor.on_train_b;£¬÷1atch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329686318 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8663322003456 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143525436 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83746686 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚573713032692736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130934612 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚9849 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143525436 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚573713014689792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329666252 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130934146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581971946709504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8229176645308 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186140816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚581972621088256 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329686318 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8259588395520 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8259588395520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47187462 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚7466.403303098 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143525438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚573713014689792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884357816979191e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329666252 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚558664328316928 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576474313593344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8299098397696 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚638157784606208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584773411991040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47412046 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576474313593344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚7501.473253468 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576474331596288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚629452303930880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605„)çº5æ$æO BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segmentÚwž· .large_pool.current‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576474313593344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚7501.473253468 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576474331596288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚629452303930880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚629452327870976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8268541111788 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576474313593344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8299098397696 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚638157784606208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584773411991040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47412046­‹Ô;2ç$- _wandbruntime‚7502  _runtime‚7502äo/ñ&è$ëM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚629513954668544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331313170 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚576530666632192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8300574201344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚561409551884288 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8299917825536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584830584457728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚629513978608640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚638220104164864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144226238 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331293104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚7502.141583474 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187086932 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144226236 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131571677 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8706149496320 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47416150 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8299917825536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84155086 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚561409534844928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _step‚814 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚576530684635136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131571234 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187086932 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331313170 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144226236?cx€ö&é$ðM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47416150 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8299917825536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84155086 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚561409534844928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _step‚814 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚576530684635136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131571234 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187086932 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331313170 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144226236 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚629513954668544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331313170 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚576530666632192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8300574201344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚561409551884288 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8299917825536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584830584457728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚629513978608640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚638220104164864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144226238 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331293104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚7502.141583474 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187086932 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144226236 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131571677 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8706149496320® wl2ê$- _wandbruntime‚7502  _runtime‚7502zÑ4e`ë$Ú /loss/FRt2/assignment_loss‚2.788025825488858 2loss/SRqqt4/assignment_loss‚1.2959758546159446 /loss/FRt3/detection_loss‚2.7385594859009696 /loss/FBt1/detection_loss‚0.6221747905986018 /loss/FBt3/detection_loss‚3.0489708364009855 0loss/FRt3/assignment_loss‚3.6835384255363826 2loss/SRqqt2/assignment_loss‚1.2944756207951145 /loss/FRt4/detection_loss‚1.9147861159096162 1loss/SRqqt2/detection_loss‚1.8041889018872206 /loss/FBt2/detection_loss‚3.7306379973888397 0loss/FRt1/assignment_loss‚1.0861501306295396 0loss/FBt2/assignment_loss‚0.2847761508392068 1loss/SRqqt3/detection_loss‚2.1409539349377154 %loss/total_loss‚40.73701098040969 trainer/global_step‚9899 2loss/SRqqt3/assignment_loss‚1.7464642327278852 epoch‚8 % _timestamp‚1.7884358174380395e+09 1loss/FBt3/assignment_loss‚0.26193429729901113 0loss/FBt4/assignment_loss‚0.5522893716374205 /loss/FBt4/detection_loss‚1.7670495841238234 0loss/FRt4/assignment_loss‚2.4894299308458963 2loss/SRqqt1/assignment_loss‚0.7163565814413301 /loss/FRt2/detection_loss‚2.4016529528833015 1loss/SRqqt1/detection_loss‚1.0078370578681366 _runtime‚7502.142686001 1loss/FBt1/assignment_loss‚0.13564796637891582 /loss/FRt1/detection_loss‚1.6674534967634826 1loss/SRqqt4/detection_loss‚1.5576814375114891 _step‚815þYeeì$ß 0loss/FRt3/assignment_loss‚3.6835384255363826 2loss/SRqqt2/assignment_loss‚1.2944756207951145 /loss/FRt4/detection_loss‚1.9147861159096162 1loss/SRqqt2/detection_loss‚1.8041889018872206 /loss/FBt2/detection_loss‚3.7306379973888397 0loss/FRt1/assignment_loss‚1.0861501306295396 0loss/FBt2/assignment_loss‚0.2847761508392068 1loss/SRqqt3/detection_loss‚2.1409539349377154 %loss/total_loss‚40.73701098040969 trainer/global_step‚9899 2loss/SRqqt3/assignment_loss‚1.7464642327278852 epoch‚8 % _timestamp‚1.7884358174380395e+09 1loss/FBt3/assignment_loss‚0.26193429729901113 0loss/FBt4/assignment_loss‚0.5522893716374205 /loss/FBt4/detection_loss‚1.7670495841238234 0loss/FRt4/assignment_loss‚2.4894299308458963 2loss/SRqqt1/assignment_loss‚0.7163565814413301 /loss/FRt2/detection_loss‚2.4016529528833015 1loss/SRqqt1/detection_loss‚1.0078370578681366 _runtime‚7502.142686001 1loss/FBt1/assignment_loss‚0.13564796637891582 /loss/FRt1/detection_loss‚1.6674534967634826 1loss/SRqqt4/detection_loss‚1.5576814375114891 _step‚815 /loss/FRt2/assignment_loss‚2.788025825488858 2loss/SRqqt4/assignment_loss‚1.2959758546159446 /loss/FRt3/detection_loss‚2.7385594859009696 /loss/FBt1/detection_loss‚0.6221747905986018 /loss/FBt3/detection_loss‚3.0489708364009855¯û}6ží$‚(:“ öºåÔʨæ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.337' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.267399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Zד&î$‚(:› öºåÔ“üù disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0) disk./.usagePercent‚73.87705059986867 disk.dm-0.in‚ 1067.51953125 disk.rbd1.out‚ 3.986328125 disk.drbd1777.in‚0$ memory_percent‚94.20926570892334* proc.memory.percent‚17.537975311279297 network.sent‚ 2.3602499e+07% disk./.usageGB‚1100.8214416503906 disk.sda1.out‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚45300.03955078125 disk.rbd0.out‚0 disk.rbd2.out‚201.1083984375 disk.sda3.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚45300.03955078125( proc.memory.availableMB‚ 948.75390625! proc.memory.rssMB‚ 2873.421875 cpu‚25.362936601941954# disk.dm-0.out‚45300.03955078125 disk.sda2.out‚0 disk.sda3.out‚297.125 disk.rbd3.out‚0 proc.cpu.threads‚17 network.recv‚ 2.554914e+06 disk.sda1.in‚0! disk.nvme0n1.in‚ 1067.51953125# disk.nvme0c0n1.in‚ 1067.51953125 disk.sda.in‚0 disk.sda.out‚297.125 disk.dm-1.out‚297.125Ž½A+žï$‚(:“ …»åÔ¶çý gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.147' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.429399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ÎÓÌð$‚(: …»åÔ‹´² disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚202.6357421875 disk.sda3.out‚297.8125 disk.rbd3.out‚0" memory_percent‚94.207763671875 disk.sda2.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.537975311279297( disk.nvme0c0n1.out‚45300.58056640625 disk.drbd1777.out‚0 disk.sda.out‚297.8125& disk.nvme0n1.out‚45300.58056640625 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1067.51953125 disk.drbd1777.in‚0 cpu‚25.36235232301717 network.recv‚ 2.561934e+06 disk.sda.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.37114e+07 disk.dm-0.in‚ 1067.51953125# disk.dm-0.out‚45300.58056640625 disk.dm-1.in‚0! disk.nvme0n1.in‚ 1067.51953125% disk./.usageGB‚1100.8215065002441 disk.rbd1.out‚ 3.986328125 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 proc.memory.availableMB‚949! proc.memory.rssMB‚ 2873.421875) disk./.usagePercent‚73.87705495199674 disk.sda1.out‚0 disk.dm-1.out‚297.8125®e2ñ$- _wandbruntime‚7537  _runtime‚7537vÈò$ % _timestamp‚1.7884358525451105e+09 _runtime‚ 7537.24989059 _step‚816 ' lr-AdamW/pg1‚2.0968411443599866e-05 ' lr-AdamW/pg2‚2.0968411443599866e-05 trainer/global_step‚9949'’VÍó$Ç ' lr-AdamW/pg1‚2.0968411443599866e-05 ' lr-AdamW/pg2‚2.0968411443599866e-05 trainer/global_step‚9949 % _timestamp‚1.7884358525451105e+09 _runtime‚ 7537.24989059 _step‚816°2ŠÎË2ô$- _wandbruntime‚7537  _runtime‚7537í…îÎS õ$áO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332907526 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚632534864754176 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚579291983538688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8309376451196 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8340084203520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132195878 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚579291965535744 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚9949 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8340740566016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47640560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47640108 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884358525458927e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_tq”ƒÜ”rain_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚579291965535744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187994504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84555318 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚641283346505216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8748481751040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚587632724104704 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144913022 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187974466 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187974466 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚817 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144913022 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187994504 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8340084203520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚572409885170680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚587632049739264 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚579291983538688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132195424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84555316 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚7537.250822145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187994504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84555318 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚641283346505216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8748481751040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚587632724104704 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144913022 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187974466 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187974466 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚817 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144913022 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187994504 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8340084203520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚572409885170680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚587632049739264 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚579291983538688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132195424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84555316 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚7537.250822145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47640560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47640108 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884358525458927e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0±FýW2÷$- _wandbruntime‚7537  _runtime‚7537׈Jð&ø$êM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚572464934259088 _step‚818 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144927036 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8341560007168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84563484 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8749166789120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚579348336577536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8310192886160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632596491551744 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚587689896584704 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188012984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188012984 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187992920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47644590 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788435853227162e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚587689896584704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚587689222205952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚9949 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47645037 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚579348318574592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8340903631360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚579348336577536 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132208523 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332940022 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚641345658340864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132208074 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚564154758412288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332919956 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚579348318574592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332940022 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚632596515491840 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844ty_!;ù$ïM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8340903631360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 332919956 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _runtime‚7537.932431656 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332940022 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚641345658340864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132208074 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚564154758412288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332919956 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚579348318574592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332940022 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚632596515491840 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚572464934259088 _step‚818 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144927036 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8341560007168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84563484 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8749166789120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚579348336577536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8310192886160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632596491551744 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚587689896584704 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188012984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_trai9.–1ºn_batch_end/allocated_bytes.all.peak‚ 9955586048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188012984 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187992920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47644590 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788435853227162e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚587689896584704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚587689222205952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚9949 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47645037 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚579348318574592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8340903631360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚579348336577536 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132208523 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84963718 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚590491361852928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145613822 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8381726371840 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188900518 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590490687487488 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188920556 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8381070009344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47869009 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334534378 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚582109617478144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8381726371840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8350211790604 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334514338 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚644408774590464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚590490687487488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚644408593317888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚635617377697280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635617401637376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8791215620608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84963716 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132832727 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚575195927038088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚575195255560972 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚590491361852928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚566845060809728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚582109617478144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145613820 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0¶D·ë'†%åO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188920556 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8381070009344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47869009 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334534378 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚582109617478144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8381726371840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8350211790604 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334514338 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚644408774590464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚590490687487488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚644408593317888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚635617377697280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635617401637376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8791215620608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84963716 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132832727 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚575195927038088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚575195255560972 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚590491361852928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚566845060809728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚582109617478144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145613820 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚582109635481088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8381070009344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84963718 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚590491361852928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145613822 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8381726371840 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188900518 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590490687487488 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334566874 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚635679052375040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚.?û¡! 671477324 % _timestamp‚1.7884358887547567e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188939036 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334546808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145627836 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8351682663532 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145627838 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188918972 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132845422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47873084 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚575251647603820 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8351028225568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚582165970516992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145627836 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚822 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚582165988519936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188918972 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84971886 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334546808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚582165970516992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚590547859954176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8381889437184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8382545812992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚575250976126496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚566899947900928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8792238052352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132844968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47873536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚590548534332928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8381889437184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚566899964940288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚644471109168128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚635679028434944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334566874 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚644471290427392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928'PÕö&‰%ðM PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132844968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47873536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚590548534332928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8381889437184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚566899964940288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚644471109168128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚635679028434944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334566874 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚644471290427392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145627838 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188939036 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚9999 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚582165988519936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334566874 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚635679052375040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 % _timestamp‚1.7884358887547567e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188939036 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334546808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145627836 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8351682663532 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145627838 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188918972 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132845422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47873084 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚575251647603820 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8351028225568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚582165970516992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145627836 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚822 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚582165988519936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188918972 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84971886 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334546808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚582165970516992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚590547859954176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8381889437184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8382545812992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚575250976126496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚566899947900928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8792238052352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8422712177664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8422712177664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚569590250298368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚584927287423488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8834289385472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚593349325235712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8391047130012 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚593349999601152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8834448815104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85372116 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚825 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚638699938520576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚593349325235712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 336161230 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146314622 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336141190 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189846608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584927287423488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚647534203965952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚638699914580480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85372118 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146314620 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚584927269420544 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚638699914580480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85372118 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146314620 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚584927269420544 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8422712177664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8422712177664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚569590250298368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚584927287423488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8834289385472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚593349325235712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8391047130012 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚593349999601152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8834448815104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85372116 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚825 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚638699938520576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚593349325235712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 336161230 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146314622 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336141190 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189846608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584927287423488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚647534203965952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133481242 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚584983622459392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85380286 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189845024 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336173660 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146328638 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _step‚826 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336193726 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48100958 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638761589258240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚584983622459392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8422875243008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146328638 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578037689471228 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336193726 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚569645154428928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚647596694887936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8392518002940 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146328636 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85380284 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚593407172081152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8422875243008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 336173660 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336173660 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146328638 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _step‚826 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336193726 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48100958 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638761589258240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚584983622459392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8422875243008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146328638 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578037689471228 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336193726 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚569645154428928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚647596694887936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8392518002940 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146328636 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85380284 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚593407172081152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8422875243008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 336173660 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133481242 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚584983622459392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85380286 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680º%j·'2œ%- _wandbruntime‚7609  _runtime‚7609Áïo[%Õ .loss/FBt2/detection_loss‚2.296319716506534 0loss/FRt1/assignment_loss‚1.3106808503468832 2loss/SRqqt1/assignment_loss‚0.9556688478504143 _runtime‚7609.100079281 /loss/FRt2/detection_loss‚2.3401104041508267 0loss/SRqqt3/detection_loss‚2.413295988783692 -loss/FBt1/detection_loss‚0.65812654059849 $loss/total_loss‚43.0107233892314 2loss/FBt3/assignment_loss‚0.009340446587884799 .loss/FRt4/detection_loss‚2.193000288033972 .loss/FBt4/detection_loss‚2.256722024508885 % _timestamp‚1.7884359243954217e+09 /loss/FBt3/detection_loss‚3.5289745926856995 /loss/FRt2/assignment_loss‚2.752952157234659 /loss/FRt3/assignment_loss‚4.291334679049831 0loss/FRt4/assignment_loss‚3.4317993971766256 1loss/SRqqt4/assignment_loss‚1.659703463781625 /loss/FRt3/detection_loss‚3.0105114540746136 1loss/SRqqt4/detection_loss‚2.0112612862949786 epoch‚8 trainer/global_step‚10049 /loss/FRt1/detection_loss‚1.3782494944830737 2loss/SRqqt2/assignment_loss‚1.4301649183034897 2loss/SRqqt3/assignment_loss‚1.9590638586969087 1loss/SRqqt2/detection_loss‚1.6341091250931774 _step‚827 1loss/FBt1/assignment_loss‚0.10741073046402012 1loss/FBt2/assignment_loss‚0.22573765303321933 1loss/FBt4/assignment_loss‚0.29763943336105775 1loss/SRqqt1/detection_loss‚0.8585460381308461?"SÓ`ž%Ú 2loss/FBt3/assignment_loss‚0.009340446587884799 .loss/FRt4/detection_loss‚2.193000288033972 .loss/FBt4/detection_loss‚2.256722024508885 % _timestamp‚1.7884359243954217e+09 /loss/FBt3/detection_loss‚3.5289745926856995 /loss/FRt2/assignment_loss‚2.752952157234659 /loss/FRt3/assignment_loss‚4.291334679049831 0loss/FRt4/assignment_loss‚3.4317993971766256 1loss/SRqqt4/assignment_loss‚1.659703463781625 /loss/FRt3/detection_loss‚3.0105114540746136 1loss/SRqqt4/detection_loss‚2.0112612862949786 epoch‚8 trainer/global_step‚10049 /loss/FRt1/detection_loss‚1.3782494944830737 2loss/SRqqt2/assignment_loss‚1.4301649183034897 2loss/SRqqt3/assignment_loss‚1.9590638586969087 1loss/SRqqt2/detection_loss‚1.6341091250931774 _step‚827 1loss/FBt1/assignment_loss‚0.10741073046402012 1loss/FBt2/assignment_loss‚0.22573765303321933 1loss/FBt4/assignment_loss‚0.29763943336105775 1loss/SRqqt1/detection_loss‚0.8585460381308461 .loss/FBt2/detection_loss‚2.296319716506534 0loss/FRt1/assignment_loss‚1.3106808503468832 2loss/SRqqt1/assignment_loss‚0.9556688478504143 _runtime‚7609.100079281 /loss/FRt2/detection_loss‚2.3401104041508267 0loss/SRqqt3/detection_loss‚2.413295988783692 -loss/FBt1/detection_loss‚0.65812654059849 $loss/total_loss‚43.0107233892314»\`ø±“Ÿ%‚(:ˆ ß»åÔ¡£ö gpu.0.gpu‚33.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚156.048' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.2096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú‡§¥% %‚(:š ß»åÔö“È network.sent‚ 2.3950582e+07 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 network.recv‚ 2.590566e+06 disk.drbd1777.out‚0 disk.rbd1.in‚0% disk.nvme0c0n1.out‚45799.94140625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1067.51953125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1067.51953125$ disk./.usageGB‚1100.823642730713 disk.sda3.out‚ 301.703125 disk.dm-0.out‚45799.94140625 disk.sda.out‚ 301.703125 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚94.21038627624512 cpu‚25.358674454227042 disk.rbd3.out‚0 disk.rbd2.in‚0! disk.rbd2.out‚204.53076171875' proc.memory.availableMB‚ 948.5703125* proc.memory.percent‚17.537975311279297 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1067.51953125) disk./.usagePercent‚73.87719831621592 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 301.703125 disk.rbd1.out‚ 3.986328125 disk.sda.in‚0# disk.nvme0n1.out‚45799.94140625! proc.memory.rssMB‚ 2873.421875Ï^ù’¡%‚(:‡ î»åÔœ¹ß gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚89.717' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9434 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Hb—ó¢%‚(:’ î»åÔëÚà network.recv‚ 2.5967e+06 disk.rbd0.out‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.537975311279297 cpu‚25.35807571441839% disk./.usageGB‚1100.8236656188965 disk.dm-1.out‚ 302.3359375 disk.sda2.out‚0 disk.sda3.out‚ 302.3359375! proc.memory.rssMB‚ 2873.421875 disk.sda.in‚0 proc.cpu.threads‚17! disk.nvme0n1.out‚ 45899.015625 disk.sda.out‚ 302.3359375 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0) disk./.usagePercent‚73.87719985226113 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚94.20888423919678 disk.dm-0.out‚ 45899.015625 disk.sda1.in‚0 disk.dm-1.in‚0! disk.nvme0n1.in‚ 1067.51953125 disk.rbd1.out‚ 3.986328125# disk.nvme0c0n1.in‚ 1067.51953125 disk.rbd0.in‚0 disk.rbd2.out‚ 204.6171875 network.sent‚ 2.4058252e+07 disk.dm-0.in‚ 1067.51953125 disk.rbd2.in‚0# disk.nvme0c0n1.out‚ 45899.015625 disk.sda3.in‚0( proc.memory.availableMB‚ 948.81640625™,Þ/2£%- _wandbruntime‚7643  _runtime‚7643ÿŒÃëʤ%Ä trainer/global_step‚10099 % _timestamp‚1.7884359592264352e+09 _runtime‚7643.931174754 _step‚828 ' lr-AdamW/pg1‚1.8211923064657532e-05 ' lr-AdamW/pg2‚1.8211923064657532e-05-B¶Ï¥%É _step‚828 ' lr-AdamW/pg1‚1.8211923064657532e-05 ' lr-AdamW/pg2‚1.8211923064657532e-05 trainer/global_step‚10099 % _timestamp‚1.7884359592264352e+09 _runtime‚7643.931174754¼µuð2¦%- _wandbruntime‚7643  _runtime‚7643†Ñ¸Aè'§%âO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚587744939365888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 337768042 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190752622 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚587744939365888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48324305 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190772660 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8431882469420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚596207962983936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8463697983488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8463041620992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8463041620992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚641782451463680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚580768010772904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚572335456826368 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8877368956928 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚10099 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190752622 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚572335473865728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚587744921362944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8877213721600 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596207962983936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85780516 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚650659844360704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147015422 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8432536907176 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8463697983488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85780518 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚580767339295788 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134105273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚829 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147015420/B$Tí'¨%çO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8463697983488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8463041620992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8463041620992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚641782451463680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚580768010772904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚572335456826368 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8877368956928 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚10099 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190752622 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚572335473865728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚587744921362944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8877213721600 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596207962983936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85780516 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚650659844360704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147015422 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8432536907176 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8463697983488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85780518 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚580767339295788 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134105273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚829 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147015420 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚587744939365888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 337768042 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190752622 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚587744939365888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48324305 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190772660 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8431882469420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚596207962983936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0½Ð*Ø42©%- _wandbruntime‚7644  _runtime‚7644ÀÀ58ûª%ëM BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚587801292404736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337820578 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚641844126141440 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147029436 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134118063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8464517424640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190771076 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650722359956992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚ 7644.60929495 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147029438 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚587801292404736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8878070204928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8878233815552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8433353342348 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚587801274401792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚596265135450624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚587801274401792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134117597 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚572390360956928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147029436 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85788684 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8464517424640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337820578 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85788686 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48328913 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337800512 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147029438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190791140 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680é‰Nö&«%ðM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147029438 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚587801292404736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8878070204928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8878233815552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8433353342348 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚587801274401792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚596265135450624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚587801274401792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134117597 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚572390360956928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147029436 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85788684 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8464517424640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337820578 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85788686 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48328913 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337800512 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147029438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190791140 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚587801292404736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337820578 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚641844126141440 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147029436 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134118063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8464517424640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190771076 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650722359956992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚ 7644.60929495 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2¾L³së2¬%- _wandbruntime‚7644  _runtime‚7644Ž•b]­%× 2loss/SRqqt1/assignment_loss‚0.8183612726967443 /loss/FBt3/detection_loss‚2.5980006796973094 trainer/global_step‚10099 0loss/FBt3/assignment_loss‚0.3454135373966502 epoch‚8 /loss/FRt4/assignment_loss‚2.928680053779057 /loss/FRt2/detection_loss‚2.1909836559341502 .loss/FRt3/detection_loss‚3.381653314051421 1loss/SRqqt3/detection_loss‚2.5618608532280756 .loss/FBt2/detection_loss‚3.196810828314887 %loss/total_loss‚40.87920395303538 1loss/FBt1/assignment_loss‚0.09798863165037011 2loss/FBt2/assignment_loss‚0.009343370737446498 /loss/FBt1/detection_loss‚0.6166050175583223 _step‚831 2loss/SRqqt3/assignment_loss‚1.6265923501602535 /loss/FRt1/detection_loss‚1.6700567705709426 0loss/SRqqt1/detection_loss‚1.150005215496728 1loss/SRqqt2/detection_loss‚1.7560459977388383 /loss/FBt4/detection_loss‚2.7604784412043437 % _timestamp‚1.7884359599060833e+09 /loss/FRt2/assignment_loss‚2.194704215113933 2loss/SRqqt2/assignment_loss‚1.3507465816289186 0loss/SRqqt4/detection_loss‚1.616555279185032 _runtime‚7644.610768126 0loss/FRt1/assignment_loss‚1.0980478017056574 2loss/SRqqt4/assignment_loss‚1.2159682776947538 /loss/FRt4/detection_loss‚2.0604547634720967 /loss/FRt3/assignment_loss‚3.511485916116963 1loss/FBt4/assignment_loss‚0.12236112790248756NAòb®%Ü /loss/FRt1/detection_loss‚1.6700567705709426 0loss/SRqqt1/detection_loss‚1.150005215496728 1loss/SRqqt2/detection_loss‚1.7560459977388383 /loss/FBt4/detection_loss‚2.7604784412043437 % _timestamp‚1.7884359599060833e+09 /loss/FRt2/assignment_loss‚2.194704215113933 2loss/SRqqt2/assignment_loss‚1.3507465816289186 0loss/SRqqt4/detection_loss‚1.616555279185032 _runtime‚7644.610768126 0loss/FRt1/assignment_loss‚1.0980478017056574 2loss/SRqqt4/assignment_loss‚1.2159682776947538 /loss/FRt4/detection_loss‚2.0604547634720967 /loss/FRt3/assignment_loss‚3.511485916116963 1loss/FBt4/assignment_loss‚0.12236112790248756 2loss/SRqqt1/assignment_loss‚0.8183612726967443 /loss/FBt3/detection_loss‚2.5980006796973094 trainer/global_step‚10099 0loss/FBt3/assignment_loss‚0.3454135373966502 epoch‚8 /loss/FRt4/assignment_loss‚2.928680053779057 /loss/FRt2/detection_loss‚2.1909836559341502 .loss/FRt3/detection_loss‚3.381653314051421 1loss/SRqqt3/detection_loss‚2.5618608532280756 .loss/FBt2/detection_loss‚3.196810828314887 %loss/total_loss‚40.87920395303538 1loss/FBt1/assignment_loss‚0.09798863165037011 2loss/FBt2/assignment_loss‚0.009343370737446498 /loss/FBt1/detection_loss‚0.6166050175583223 _step‚831 2loss/SRqqt3/assignment_loss‚1.6265923501602535¿‚©{‘¯%‚(:† ý»åԜ߉ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.97' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;bïÛ,°%‚(:¡ ý»åԌץ network.sent‚ 2.4064631e+07( disk./.usagePercent‚73.8771921720351 disk.rbd2.out‚ 204.6328125# disk.dm-0.out‚45900.04833984375 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125% proc.memory.availableMB‚ 948.78125% disk./.usageGB‚1100.8235511779785 disk.sda3.in‚0 disk.drbd1777.out‚0 cpu‚25.3574699388197 disk.rbd3.out‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚45900.04833984375* proc.memory.percent‚17.537975311279297 network.recv‚ 2.601705e+06 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 302.77734375 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 1067.51953125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.out‚ 302.77734375& disk.nvme0n1.out‚45900.04833984375# disk.nvme0c0n1.in‚ 1067.51953125 disk.sda2.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 1067.51953125 disk.sda.in‚0 disk.dm-1.out‚ 302.77734375$ memory_percent‚94.20909881591797! proc.memory.rssMB‚ 2873.421875Ž¿l “±%‚(:ˆ Œ¼åÔü»ŒŽ gpu.0.gpu‚41.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†YÃS)²%‚(:ž Œ¼åÔü¬” disk.drbd1777.in‚0 disk.drbd1777.out‚0 network.sent‚ 2.4172627e+07 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0# disk.dm-0.out‚45999.08349609375 disk.rbd1.in‚0 cpu‚25.356871249719628% disk./.usageGB‚1100.8236503601074 disk.sda1.out‚0! proc.memory.rssMB‚ 2873.421875 disk.sda.out‚ 303.53125( disk.nvme0c0n1.out‚45999.08349609375 disk.sda2.out‚0( proc.memory.availableMB‚ 948.75390625* proc.memory.percent‚17.537975311279297 disk.sda3.out‚ 303.53125 disk.sda1.in‚0 disk.rbd1.out‚ 3.986328125# disk.nvme0c0n1.in‚ 1067.51953125$ memory_percent‚94.20926570892334 network.recv‚ 2.607784e+06 disk.rbd3.out‚0 disk.dm-0.in‚ 1067.51953125 proc.cpu.threads‚17 disk.rbd2.out‚ 204.6953125 disk.rbd3.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 303.53125! disk.nvme0n1.in‚ 1067.51953125& disk.nvme0n1.out‚45999.08349609375 disk.sda2.in‚0) disk./.usagePercent‚73.87719882823099äh’P2³%- _wandbruntime‚7679  _runtime‚7679‡{ªÄ´%¾ trainer/global_step‚10149 % _timestamp‚1.7884359951143453e+09 _runtime‚7679.819068663 _step‚832 $ lr-AdamW/pg1‚1.7332479546449e-05 $ lr-AdamW/pg2‚1.7332479546449e-05 ƒ““ɵ%à $ lr-AdamW/pg1‚1.7332479546449e-05 $ lr-AdamW/pg2‚1.7332479546449e-05 trainer/global_step‚10149 % _timestamp‚1.7884359951143453e+09 _runtime‚7679.819068663 _step‚832Àxñ92¶%- _wandbruntime‚7679  _runtime‚7679+y¡zè'·%âO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚469 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134742082 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚590562573305344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8504683789312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚653785284376576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚583553381163196 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134742551 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8504027426816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191678674 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147716222 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86188918 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚653785090521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884359951150453e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147716220 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86188916 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575080663354368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8504683789312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚599067275097600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚467 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚833 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚599066600732160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚590562591308288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191698712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚590562573305344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644865012286976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191698712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚599066600732160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339394894 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147716222 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191678674 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8473372246584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7679.819947108 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147716220 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147716220 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86188916 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575080663354368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8504683789312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚599067275097600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚467 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚833 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚599066600732160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚590562591308288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191698712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚590562573305344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644865012286976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191698712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚599066600732160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339394894 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147716222 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191678674 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8473372246584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7679.819947108 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147716220Áwf%‚(:ˆ ›¼åÔ±­„Ž gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.258' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.4516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê.º%‚(:£ ›¼åÔÑ¢ä‘# disk.nvme0c0n1.in‚ 1067.51953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 304.5234375 disk.rbd3.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 206.22265625 disk.dm-0.in‚ 1067.51953125 disk.dm-1.out‚ 304.5234375( proc.memory.availableMB‚ 948.51171875) disk./.usagePercent‚73.87719652416318 disk.rbd0.in‚0 disk.sda3.in‚0 network.sent‚ 2.4179004e+07 network.recv‚ 2.612435e+06 disk.drbd1777.in‚0 disk.drbd1777.out‚0& disk.nvme0n1.out‚46098.83154296875 disk.rbd1.out‚ 3.986328125! proc.memory.rssMB‚ 2873.421875 proc.cpu.threads‚17( disk.nvme0c0n1.out‚46098.83154296875 disk.sda.in‚0 disk.sda.out‚ 304.5234375 disk.sda1.out‚0$ memory_percent‚94.21074390411377 disk.rbd1.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚1100.823616027832! disk.nvme0n1.in‚ 1067.51953125 disk.rbd3.in‚0# disk.dm-0.out‚46098.83154296875* proc.memory.percent‚17.537975311279297 cpu‚25.35626279811181KØ»2»%- _wandbruntime‚7680  _runtime‚7680ÎAXò&¼%ìM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚644926639084544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339447430 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚599124447577600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8504846854656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _runtime‚7680.488853118 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339427364 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191697128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8473534243792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339447430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8920958282752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8505503230464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48558262 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚575135567484928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚590618926344192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚465 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚575135584524288 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644926663024640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134755348 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚590618944347136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8504846854656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚590618926344192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 339427364 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚583609773206044 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884359957836838e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134754881 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚10149 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚599123773198848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86197086 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191717192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86197084 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚599124447577600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚834 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8505503230464 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚467 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚653847597367296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚599124447577600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8504846854656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _runtime‚7680.488853118 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339427364 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191697128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8473534243792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339447430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8920958282752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8505503230464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48558262 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚575135567484928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚590618926344192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚465 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚575135584524288 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644926663024640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134755348 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚590618944347136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8504846854656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚590618926344192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 339427364 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚583609773206044 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884359957836838e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134754881 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚10149 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚599123773198848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86197086 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191717192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86197084 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚599124447577600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚834 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8505503230464 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚467 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚653847597367296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Â×Z3d2¾%- _wandbruntime‚7680  _runtime‚7680¥¡<íY¿%Ó 0loss/SRqqt2/detection_loss‚1.666949522392503 epoch‚8 2loss/SRqqt1/assignment_loss‚1.0074511361759035 .loss/FRt4/detection_loss‚2.031475907023906 0loss/SRqqt1/detection_loss‚1.121097642207338 /loss/FBt1/detection_loss‚0.4856100476650421 -loss/FBt2/detection_loss‚2.27933726140431 _step‚835 /loss/FRt1/detection_loss‚1.8331292459839268 1loss/SRqqt4/detection_loss‚1.9969459791132744 % _timestamp‚1.7884359957856266e+09 /loss/FRt4/assignment_loss‚3.123663499138572 2loss/SRqqt3/assignment_loss‚1.5080267454031855 .loss/FRt3/detection_loss‚2.871994243065516 .loss/FBt4/detection_loss‚2.184869945049286 0loss/FBt1/assignment_loss‚0.0948536659678265 1loss/FBt3/assignment_loss‚0.29985165879285586 %loss/total_loss‚39.57548230623245 _runtime‚7680.490254158 1loss/SRqqt3/detection_loss‚2.5406565833836794 0loss/FRt1/assignment_loss‚1.1566995005858571 0loss/FRt3/assignment_loss‚2.5026830712954204 0loss/FBt2/assignment_loss‚0.2732083620472362 trainer/global_step‚10149 0loss/FRt2/assignment_loss‚2.9323029769791495 2loss/SRqqt2/assignment_loss‚1.0587914735399957 .loss/FRt2/detection_loss‚2.406955427924792 /loss/FBt3/detection_loss‚2.5304123096995883 2loss/SRqqt4/assignment_loss‚1.5049887258321681 1loss/FBt4/assignment_loss‚0.16352737556111865ÜñY8ÕÀ%Ø .loss/FRt2/detection_loss‚2.406955427924792 /loss/FBt3/detection_loss‚2.5304123096995883 2loss/SRqqt4/assignment_loss‚1.5049887258321681 1loss/FBt4/assignment_loss‚0.16352737556111865 0loss/SRqqt2/detection_loss‚1.666949522392503 epoch‚8 2loss/SRqqt1/assignment_loss‚1.0074511361759035 .loss/FRt4/detection_loss‚2.031475907023906 0loss/SRqqt1/detection_loss‚1.121097642207338 /loss/FBt1/detection_loss‚0.4856100476650421 K;x‰-loss/FBt2/detection_loss‚2.27933726140431 _step‚835 /loss/FRt1/detection_loss‚1.8331292459839268 1loss/SRqqt4/detection_loss‚1.9969459791132744 % _timestamp‚1.7884359957856266e+09 /loss/FRt4/assignment_loss‚3.123663499138572 2loss/SRqqt3/assignment_loss‚1.5080267454031855 .loss/FRt3/detection_loss‚2.871994243065516 .loss/FBt4/detection_loss‚2.184869945049286 0loss/FBt1/assignment_loss‚0.0948536659678265 1loss/FBt3/assignment_loss‚0.29985165879285586 %loss/total_loss‚39.57548230623245 _runtime‚7680.490254158 1loss/SRqqt3/detection_loss‚2.5406565833836794 0loss/FRt1/assignment_loss‚1.1566995005858571 0loss/FRt3/assignment_loss‚2.5026830712954204 0loss/FBt2/assignment_loss‚0.2732083620472362 trainer/global_step‚10149 0loss/FRt2/assignment_loss‚2.9323029769791495 2loss/SRqqt2/assignment_loss‚1.0587914735399957Ã…5,¬œÁ%‚(:‘ ª¼åÔ¯äŽ gpu.0.gpu‚40.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚99.53' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.906000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òIåü/Â%‚(:¤ ª¼åÔ¶ââ disk.dm-0.in‚ 1067.51953125 disk.dm-1.out‚ 305.05078125 cpu‚25.355674719428556) disk./.usagePercent‚73.87726513418235 disk.rbd0.out‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 1067.51953125 disk.sda2.out‚0 network.recv‚ 2.618456e+06% disk./.usageGB‚1100.8246383666992 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 206.28515625$ proc.memory.availableMB‚948.6875 disk.drbd1777.out‚0# disk.dm-0.out‚46197.97216796875 disk.sda.in‚0 disk.sda.out‚ 305.05078125! disk.nvme0n1.in‚ 1067.51953125! proc.memory.rssMB‚ 2873.421875* proc.memory.percent‚17.537975311279297 network.sent‚ 2.4275472e+07& disk.nvme0n1.out‚46197.97216796875 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 305.05078125 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚46197.97216796875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 3.986328125$ memory_percent‚94.20967102050781 proc.cpu.threads‚17‘øYÃ%‚(:’ ¹¼åԢˑŽ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.839' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.767799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘´, /Ä%‚(:¤ ¹¼åÔ¢‡Ù disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 cpu‚25.35508251187373 disk.sda.in‚0# disk.dm-0.out‚46297.12060546875! disk.nvme0n1.in‚ 1067.51953125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1067.51953125( disk.nvme0c0n1.out‚46297.12060546875 disk.rbd0.in‚0 disk.rbd2.out‚ 206.30078125$ memory_percent‚94.20969486236572) disk./.usagePercent‚73.87726334212962 disk.sda3.out‚ 305.6484375 disk.drbd1777.in‚0 disk.dm-0.in‚ 1067.51953125 disk.dm-1.out‚ 305.6484375& disk.nvme0n1.out‚46297.12060546875% disk./.usageGB‚1100.8246116638184 disk.rbd3.in‚0 disk.sda.out‚ 305.6484375 disk.rbd1.out‚ 3.986328125! proc.memory.rssMB‚ 2873.421875 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 network.sent‚ 2.4373625e+07( proc.memory.availableMB‚ 948.68359375* proc.memory.percent‚17.537975311279297 network.recv‚ 2.624615e+06ŠBpÔ2Å%- _wandbruntime‚7715  _runtime‚7715F\S8ÈÆ% & lr-AdamW/pg2‚1.647303789210604e-05 trainer/global_step‚10199 % _timestamp‚1.7884360306529088e+09 _runtime‚7715.357709337 _step‚836 & lr-AdamW/pg1‚1.647303789210604e-05ìè—ÍÇ%Ç & lr-AdamW/pg2‚1.647303789210604e-05 trainer/global_step‚10199 % _timestamp‚1.7884360306529088e+09 _runtime‚7715.357709337 _step‚836 & lr-AdamW/pg1‚1.647303789210604e-05Ķ<ª2È%- _wandbruntime‚7715  _runtime‚7715¾ÐÈè'É%âO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8514207585992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚601925912845824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135379688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192624764 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48782370 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚647947549170176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚577825886921728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚586339423030604 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚656910518214144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚647947525230080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884360306536548e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8513553148236 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚593380243250688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8545013232640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚593380225247744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86597318 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148417022 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚601925912845824 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192604726 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚586340094507720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚577825869882368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192604726 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148417020 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8545013232640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚601925238480384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341021746 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8545669595136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8963154510848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8962992984064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚601925238480384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚593380225247744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚593380243250688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _runtime‚7715.358540423 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341021746 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8545669595136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8963154510848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8962992984064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚601925238480384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚593380225247744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚593380243250688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _runtime‚7715.358540423 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135379688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192624764 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48782370 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚647947549170176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚577825886921728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚586339423030604 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚656910518214144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚647947525230080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884360306536548e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8513553148236 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚593380243250688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8545013232640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚593380225247744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86597318 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148417022 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚601925912845824 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192604726 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚586340094507720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚577825869882368Å ÝÖ2Ë%- _wandbruntime‚7716  _runtime‚77168iäm…Ì%ìM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48786422 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚593436596289536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135392349 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚648009199907840 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341074282 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577880774012928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8545832660480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚656973212674560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148431038 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8964012766720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8963842864640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚586395143596128 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148431036 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86605484 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚586395815073452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚593436578286592 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192623180 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135391906 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚656973018832384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577880791052288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884360313285992e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341054216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 341074282 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚10199 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚601982410947072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8546489036288 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148431036 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192643244 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚593436596289536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 _runtime‚7716.033842782 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192623180 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0óê¤m÷&Í%ñM E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚601982410947072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8546489036288 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148431036 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192643244 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚593436596289536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 _runtime‚7716.033842782 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192623180 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48786422 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚593436596289536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135392349 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚648009199907840 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341074282 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577880774012928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8545832660480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚656973212674560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148431038 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8964012766720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8963842864640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚586395143596128 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148431036 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86605484 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚586395815073452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚593436578286592 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192623180 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135391906 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚656973018832384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577880791052288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884360313285992e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341054216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 341074282 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚10199 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ÆD® 2Î%- _wandbruntime‚7716  _runtime‚7716}ˆÉZÏ%Ô 0loss/FRt4/assignment_loss‚2.6480923034250736 /loss/FBt2/detection_loss‚2.9186399221420287 trainer/global_step‚10199 2loss/SRqqt2/assignment_loss‚1.0408355501689472 1loss/FBt3/assignment_loss‚0.12834006683745733 0loss/FBt4/assignment_loss‚0.5043672436316099 /loss/FRt1/detection_loss‚1.4630067283814807 % _timestamp‚1.7884360313302937e+09 _step‚839 /loss/FRt1/assignment_loss‚1.635127528658453 0loss/FRt3/assignment_loss‚3.4453779481924496 2loss/SRqqt1/assignment_loss‚1.1050344039686024 1loss/SRqqt3/assignment_loss‚1.736654983445381 2loss/SRqqt4/assignment_loss‚1.5491641486684482 0loss/FBt1/assignment_loss‚0.0653337983062859 1loss/SRqqt1/detection_loss‚0.7087410072631696 /loss/SRqqt3/detection_loss‚2.74355390916268 1loss/SRqqt4/detection_loss‚1.6673884750516326 %loss/total_loss‚42.03381879633255 .loss/FRt2/detection_loss‚2.529336664734817 /loss/FBt1/detection_loss‚0.6907531174830284 .loss/FBt3/detection_loss‚2.915733724832535 _runtime‚ 7716.03496118 /loss/FBt4/detection_loss‚3.1653115834508623 1loss/FBt2/assignment_loss‚0.12164742820750689 1loss/SRqqt2/detection_loss‚1.7037917135321363 epoch‚8 /loss/FRt3/detection_loss‚2.7250589315707865 .loss/FRt4/detection_loss‚2.263763912022114 0loss/FRt2/assignment_loss‚2.5587637031950603Ò@æF_Ð%Ù /loss/FRt1/assignment_loss‚1.635127528658453 0loss/FRt3/assignment_loss‚3.4453779481924496 2loss/SRqqt1/assignment_loss‚1.1050344039686024 1loss/SRqqt3/assignment_loss‚1.736654983445381 2loss/SRqqt4/assignment_loss‚1.5491641486684482 0loss/FBt1/assignment_loss‚0.0653337983062859 1loss/SRqqt1/detection_loss‚0.7087410072631696 /loss/SRqqt3/detection_loss‚2.74355390916268 1loss/SRqqt4/detection_loss‚1.6673884750516326 %loss/total_loss‚42.03381879633255 .loss/FRt2/detection_loss‚2.529336664734817 /loss/FBt1/detection_loss‚0.6907531174830284 .loss/FBt3/detection_loss‚2.915733724832535 _runtime‚ 7716.03496118 /loss/FBt4/detection_loss‚3.1653115834508623 1loss/FBt2/assignment_loss‚0.12164742820750689 1loss/SRqqt2/detection_loss‚1.7037917135321363 epoch‚8 /loss/FRt3/detection_loss‚2.7250589315707865 .loss/FRt4/detection_loss‚2.263763912022114 0loss/FRt2/assignment_loss‚2.5587637031950603 0loss/FRt4/assignment_loss‚2.6480923034250736 /loss/FBt2/detection_loss‚2.9186399221420287 trainer/global_step‚10199 2loss/SRqqt2/assignment_loss‚1.0408355501689472 1loss/FBt3/assignment_loss‚0.12834006683745733 0loss/FBt4/assignment_loss‚0.5043672436316099 /loss/FRt1/detection_loss‚1.4630067283814807 % _timestamp‚1.7884360313302937e+09 _step‚839Ç+fc Ñ%‚(:’ ȼåÔê¶ç gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.45' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.889999999999993 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äâ-N+Ò%‚(:  ȼåÔü¼¨ network.recv‚ 2.628815e+06 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 306.140625 disk.drbd1777.out‚0$ memory_percent‚94.20990943908691 cpu‚25.35449468939778 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.537975311279297 network.sent‚ 2.4379104e+07 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1100.8246688842773# disk.nvme0c0n1.in‚ 1067.51953125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 2873.421875 disk.sda2.out‚0 disk.sda3.in‚0# disk.dm-0.out‚46297.42919921875( disk.nvme0c0n1.out‚46297.42919921875 disk.rbd2.out‚ 206.30078125' proc.memory.availableMB‚ 948.6484375 disk.dm-0.in‚ 1067.51953125 disk.sda.out‚ 306.140625) disk./.usagePercent‚73.87726718224263 disk.dm-1.out‚ 306.140625 disk.rbd1.out‚ 3.986328125! disk.nvme0n1.in‚ 1067.51953125& disk.nvme0n1.out‚46297.42919921875ˆ±Bp’Ó%‚(:‡ ×¼åÔŽû´Ž gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚90.143' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0286 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹62Ô%‚(:§ ×¼åÔ쥱‘ disk.sda.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚46396.74560546875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0' proc.memory.availableMB‚ 948.1640625! proc.memory.rssMB‚ 2873.421875 network.recv‚ 2.635834e+06 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 1067.51953125 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0! disk.rbd2.out‚206.40283203125 disk.rbd3.in‚0 cpu‚25.353899206436086% disk./.usageGB‚1100.8246116638184 disk.rbd1.in‚0# disk.dm-0.out‚46396.74560546875$ memory_percent‚94.21286582946777 network.sent‚ 2.4487936e+07 disk.sda2.in‚0 disk.sda3.out‚ 306.6953125 disk.drbd1777.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.537975311279297! disk.nvme0n1.in‚ 1067.51953125& disk.nvme0n1.out‚46396.74560546875 disk.sda2.out‚0 disk.drbd1777.out‚0) disk./.usagePercent‚73.87726334212962 disk.sda.out‚ 306.6953125 disk.rbd1.out‚ 3.986328125 disk.dm-0.in‚ 1067.51953125 disk.dm-1.out‚ 306.6953125pÚs2Õ%- _wandbruntime‚7751  _runtime‚7751E`˜ôÉÖ%à ' lr-AdamW/pg1‚1.5633777207981296e-05 ' lr-AdamW/pg2‚1.5633777207981296e-05 trainer/global_step‚10249 $ _timestamp‚1.788436066820534e+09 _runtime‚7751.525261006 _step‚840L)a Î×%È ' lr-AdamW/pg1‚1.5633777207981296e-05 ' lr-AdamW/pg2‚1.5633777207981296e-05 trainer/global_step‚10249 $ _timestamp‚1.788436066820534e+09 _runtime‚7751.525261006 _step‚840ÈŠ„– 2Ø%- _wandbruntime‚7751  _runtime‚7751õ€Ìšè'Ù%âO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193530778 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9006032251392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87005718 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚596197895193088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8586655400960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚580571076410368 trainer/global_step‚10249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596197895193088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193530778 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚604784550594048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8555042925400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149117822 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8585999038464 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149117820 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚430 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _step‚841 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149117822 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136016694 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚596197877190144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660036271442944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8586655400960 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193550816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚589126136375128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8554388487644 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149117820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193550816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚596197877190144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚580571093449728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚7751.526153808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8585999038464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚604784550594048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚651030062113280 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604783876228608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87005716 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9006185389568 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49010978 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8586655400960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚580571076410368 trainer/global_step‚10249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596197895193088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193530778 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚604784550594048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8555042925400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149117822 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8585999038464 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149117820 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚430 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 86822:'00928 _step‚841 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149117822 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136016694 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚596197877190144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660036271442944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8586655400960 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193550816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚589126136375128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8554388487644 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149117820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193550816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚596197877190144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚580571093449728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚7751.526153808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8585999038464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚604784550594048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚651030062113280 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604783876228608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87005716 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9006185389568 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49010978 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660098584563712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚604841723074048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136029252 trainer/global_step‚10249 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8586818466304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193569296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193549232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651091712850944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚580625997580288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49015822 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596254248231936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚596254230228992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149131838 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884360675115137e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342681068 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87013886 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660098782600192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚589181185463536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚842 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8587474842112 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149131836 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚589181856940860 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87013884 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚580625980540928 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚604841048695296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8587474842112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚7752.216930668 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8586818466304 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136029708 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342701134N`)÷&Ý%ñM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚651091736791040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9007045809152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9006871712768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342681068 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚604841723074048 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149131836 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87013884 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚580625980540928 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚604841048695296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8587474842112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚7752.216930668 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8586818466304 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136029708 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342701134 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49015368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660098584563712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚604841723074048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136029252 trainer/global_step‚10249 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8586818466304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193569296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193549232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651091712850944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚580625997580288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49015822 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596254248231936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚596254230228992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149131838 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884360675115137e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342681068 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87013886 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660098782600192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚589181185463536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚842 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8587474842112 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149131836 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚589181856940860 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8627641206784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚663161656491008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 194456830 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149818622 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194476868 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8626984844288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607642513976832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚7787.077522886 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚654112622936576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚591911506765420 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607642513976832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚7787.077522886 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚654112622936576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚591911506765420 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8627641206784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚663161656491008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 194456830 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149818622 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194476868 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8626984844288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040Í.·³Ð2í%- _wandbruntime‚7787  _runtime‚7787dXÄò&î%ìM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591967898808268 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8627804272128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49242431 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 % _timestamp‚1.7884361030420024e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8628460647936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149832636 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87422286 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚607700360822272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9050081957888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149832636 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136665169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194475284 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚599071882171392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚663224172275712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9049922541568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚10299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚599071900174336 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344327986 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚654174249734144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344327986 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344307920 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136664715 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194475284 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048õÖöï%ñM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚607699686443520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8596040262016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMoOhn#nitor.on_train_batch_end/allocation.all.allocated‚ 344327986 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚654174249734144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344327986 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344307920 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136664715 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194475284 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591967898808268 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8627804272128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49242431 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 % _timestamp‚1.7884361030420024e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8628460647936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149832636 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87422286 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚607700360822272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9050081957888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149832636 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136665169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194475284 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚599071882171392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚663224172275712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9049922541568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚10299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚599071900174336 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 345902302 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8636059166460 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7822.786970352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150519420 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87822518 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150519422 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8667970650112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195382882 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150519422 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚610501826090496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137289717 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚657195135879680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884361380820312e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚610501826090496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚586061489466368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345922342 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137290175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87822516 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚666287368730624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8636713604216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610501151725056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9092232850944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚601833199077888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150519420 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚594697548632828 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8668627012608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195402920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666287562586112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8668627012608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9092402766336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0L¹T!Hü%çO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚610501826090496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚586061489466368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345922342 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137290175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87822516 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚666287368730624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8636713604216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610501151725056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9092232850944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚601833199077888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150519420 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚594697548632828 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8668627012608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195402920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666287562586112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8668627012608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9092402766336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚10349 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚601833181074944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚601833181074944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610501151725056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49467657 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 345902302 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8636059166460 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 @8DeviceStatsMonitor.oÁîÐè¥ n_train_batch_start/num_device_alloc‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7822.786970352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150519420 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87822518 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150519422 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8667970650112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195382882 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150519422 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚610501826090496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137289717 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚657195135879680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884361380820312e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872Ñïû“í2ý%- _wandbruntime‚7823  _runtime‚7823)©ØŒò&þ%ìM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150533438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150533436 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137302878 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87830686 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137302427 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚850 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150533438 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49472192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9093103303680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150533436 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚586116410636288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _runtime‚7823.448538453 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345954838 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195401336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚601889552116736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚610558324191744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8669446453760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚657256810557440 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87830684 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚594753940675676 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666349889921024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚601889552116736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚610558998570496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657256786617344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8668790077952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚594753269198352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345954838 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚601889534113792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9093260622848 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚601889534113792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 345934772 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195421400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8637530039388 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884361387431688e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195421400 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚610558998570496ê÷&ÿ%ñM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87830684 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚594753940675676 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666349889921024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚601889552116736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚610558998570496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657256786617344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8668790077952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚594753269198352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345954838 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚601889534113792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9093260622848 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚601889534113792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 345934772 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195421400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8637530039388 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884361387431688e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195421400 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚610558998570496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150533438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150533436 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137302878 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87830686 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137302427 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚850 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150533438 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49472192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9093103303680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150533436 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚586116410636288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _runtime‚7823.448538453 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345954838 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195401336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚601889552116736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚610558324191744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8669446453760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚657256810557440 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2Òø#Ш2€&- _wandbruntime‚7823  _runtime‚78238T‹s\&Ö /loss/FBt2/detection_loss‚1.8904418105428868 0loss/FRt1/assignment_loss‚1.1549212499098345 /loss/FRt3/assignment_loss‚2.995939925312996 1loss/FBt2/assignment_loss‚0.32488454844728537 1loss/FBt3/assignment_loss‚0.31084787063425917 .loss/FBt4/detection_loss‚2.182441453138987 _step‚851 /loss/FRt2/assignment_loss‚2.904587527927087 2loss/SRqqt1/assignment_loss‚0.9649035700766201 1loss/FBt4/assignment_loss‚0.49829204018331236 epoch‚8 $ _timestamp‚1.788436138745059e+09 0loss/FBt1/assignment_loss‚0.1663360623996029 /loss/FRt2/detection_loss‚2.1421138413098393 /loss/FBt3/detection_loss‚2.8461842272016735 _runtime‚7823.450020185 /loss/FRt4/detection_loss‚2.2549646146919415 1loss/SRqqt2/detection_loss‚1.6892592080614783 0loss/FBt1/detection_loss‚0.48377119840558946 2loss/SRqqt2/assignment_loss‚1.2669679106297818 /loss/FRt1/detection_loss‚1.5816410045732152 0loss/SRqqt4/detection_loss‚1.668336613310708 1loss/SRqqt1/detection_loss‚0.9897037156377778 trainer/global_step‚10349 2loss/SRqqt4/assignment_loss‚1.5241159182731752 .loss/FRt3/detection_loss‚3.380075345436732 %loss/total_loss‚40.38703316468329 /loss/FRt4/assignment_loss‚2.793813185847324 1loss/SRqqt3/assignment_loss‚1.730829252187034 1loss/SRqqt3/detection_loss‚2.6416610705441443 a¿DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613359789473280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151220220 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588806713033728 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151220220 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚613360463838720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚604650833017344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137926900 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚604650833017344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347529154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196328972 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚604650851020288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚597483590500236 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8676894505868 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884361734958441e+09RÇXí'Œ&çO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚669412943074304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196308934 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151220222 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137926444 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49695528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8709612818432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚660277696702976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚853 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8708956455936 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151220222 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137926900 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚604650833017344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347529154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196328972 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚604650851020288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚597483590500236 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8676894505868 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884361734958441e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8709612818432 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196328972 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88230918 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49695982 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚10399 _runtime‚7858.200840713 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚613360463838720 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196308934 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88230916 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588806695994368 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613359789473280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151220220 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588806713033728 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151220220 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚613360463838720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚604650833017344 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137939082 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347581690 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604707204059136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196347452 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196347452 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚604707186056192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚669475469616128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚597539982543084 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9136146115584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9136303434752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597539311065760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151234238 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196327388 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137939550 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚613417636318720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚660339347440640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _step‚854 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151234236 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚468 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347581690 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49700464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8710432259584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347561624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚604707204059136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151234236 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8709775883776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚882390845ë D&ñM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347561624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚604707204059136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151234236 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8709775883776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88239084 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚588861600124928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137939082 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347581690 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604707204059136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196347452 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196347452 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚604707186056192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚669475469616128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚597539982543084 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9136146115584 UHDe¤údųviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9136303434752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597539311065760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151234238 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196327388 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137939550 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚613417636318720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚660339347440640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _step‚854 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151234236 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚468 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347581690 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49700464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8710432259584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844Ö:¦ ¡2&- _wandbruntime‚7859  _runtime‚7859lçž›\‘&Ö 2loss/FBt3/assignment_loss‚0.008429147049810126 1loss/SRqqt4/detection_loss‚1.4948405865579844 /loss/FBt4/detection_loss‚2.9617561250925064 0loss/FRt1/assignment_loss‚1.0737644043835726 .loss/FRt4/detection_loss‚1.782590934560328 1loss/SRqqt2/detection_loss‚1.6084137061430561 .loss/FBt2/detection_loss‚2.604488396644592 ,loss/FBt3/detection_loss‚4.9538494348526 % _timestamp‚1.7884361743763702e+09 0loss/FRt3/assignment_loss‚2.8981591446532144 &loss/total_loss‚41.557870713292004 trainer/global_step‚10399 2loss/SRqqt4/assignment_loss‚1.0555877417864394 .loss/FRt3/detection_loss‚2.829046602050463 _step‚855 /loss/FRt2/assignment_loss‚2.482780646710169 2loss/SRqqt1/assignment_loss‚0.8147719935805071 1loss/FBt2/assignment_loss‚0.31005699770175854 /loss/FRt1/detection_loss‚1.4408013408834284 0loss/SRqqt3/detection_loss‚2.553236000239849 /loss/FBt1/detection_loss‚0.7746145226393959 _runtime‚7859.080981316 .loss/FRt2/detection_loss‚2.648080200666473 1loss/SRqqt1/detection_loss‚1.0168172363482881 epoch‚8 2loss/SRqqt3/assignment_loss‚1.7522659885386627 1loss/FBt4/assignment_loss‚0.13937365973833948 0loss/FRt4/assignment_loss‚2.7437998806988753 1loss/SRqqt2/assignment_loss‚1.528972378180873 1loss/FBt1/assignment_loss‚0.08137364359082011"ZÚàa’&Û 1loss/SRqqt1/detection_loss‚1.0168172363482881 epoch‚8 2loss/SRqqt3/assignment_loss‚1.7522659885386627 1loss/FBt4/assignment_loss‚0.13937365973833948 0loss/FRt4/assignment_loss‚2.7437998806988753 1loss/SRqqt2/assignment_loss‚1.528972378180873 1loss/FBt1/assignment_loss‚0.08137364359082011 2loss/FBt3/assignment_loss‚0.008429147049810126 1loss/SRqqt4/detection_loss‚1.4948405865579844 /loss/FBt4/detection_loss‚2.9617561250925064 0loss/FRt1/assignment_loss‚1.0737644043835726 .loss/FRt4/detection_loss‚1.782590934560328 1loss/SRqqt2/detection_loss‚1.6084137061430561 .loss/FBt2/detection_loss‚2.604488396644592 ,loss/FBt3/detection_loss‚4.9538494348526 % _timestamp‚1.7884361743763702e+09 0loss/FRt3/assignment_loss‚2.8981591446532144 &loss/total_loss‚41.557870713292004 trainer/global_step‚10399 2loss/SRqqt4/assignment_loss‚1.0555877417864394 .loss/FRt3/detection_loss‚2.829046602050463 _step‚855 /loss/FRt2/assignment_loss‚2.482780646710169 2loss/SRqqt1/assignment_loss‚0.8147719935805071 1loss/FBt2/assignment_loss‚0.31005699770175854 /loss/FRt1/detection_loss‚1.4408013408834284 0loss/SRqqt3/detection_loss‚2.553236000239849 /loss/FBt1/detection_loss‚0.7746145226393959 _runtime‚7859.080981316 .loss/FRt2/detection_loss‚2.648080200666473×~3DÌ““&‚(:ˆ ϽåÔûÍûŽ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.361' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.8722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾öü3”&‚(:¨ ϽåÔîàÖ# disk.dm-0.out‚46993.07861328125 disk.drbd1777.in‚0! proc.memory.rssMB‚ 2873.421875 disk.sda.out‚ 311.33984375 disk.dm-1.out‚ 311.33984375 disk.rbd1.in‚0 disk.sda3.out‚ 311.33984375 disk.rbd3.out‚0$ memory_percent‚94.21200752258301 cpu‚25.34924789796891! disk.nvme0n1.in‚ 1067.51953125& disk.nvme0n1.out‚46993.07861328125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 3.986328125( disk.nvme0c0n1.out‚46993.07861328125 disk.rbd0.out‚0* proc.memory.percent‚17.537975311279297) disk./.usagePercent‚73.87727972661182% disk./.usageGB‚1100.8248558044434 disk.dm-0.in‚ 1067.51953125# disk.nvme0c0n1.in‚ 1067.51953125 network.sent‚ 2.4842539e+07 network.recv‚ 2.676928e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 948.3046875 disk.rbd2.out‚209.8291015625 disk.drbd1777.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17i’•&‚(:‡ Þ½åÔ”äî gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.893' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9786 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S¥,d–&‚(:“ Þ½åÔÔ¨¤‘$ disk.nvme0n1.out‚47092.259765625 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 1067.5703125 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 311.96875 disk.dm-1.in‚0% proc.memory.availableMB‚ 947.96875 disk.rbd2.out‚209.8447265625 disk.sda.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.537975311279297 cpu‚25.34867584224105 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 311.96875 disk.rbd3.in‚0! proc.memory.rssMB‚ 2873.421875 network.sent‚ 2.4950351e+07 disk.rbd1.out‚ 3.986328125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1067.5703125 disk.dm-1.out‚ 311.96875 network.recv‚ 2.682873e+06) disk./.usagePercent‚73.87728331071729& disk.nvme0c0n1.out‚47092.259765625 disk.drbd1777.in‚0! disk.dm-0.out‚47092.259765625$ disk./.usageGB‚1100.824909210205 disk.nvme0n1.in‚ 1067.5703125 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0$ memory_percent‚94.21405792236328ÉŽkž—&‚(:“ í½åÔƒêâ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.329' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.665800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷®M¶2˜&‚(:§ í½åÔáÊÓ disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0* proc.memory.percent‚17.537975311279297 network.recv‚ 2.687535e+06 disk.rbd2.in‚0! disk.rbd2.out‚209.93115234375 disk.rbd3.in‚0( proc.memory.availableMB‚ 947.78515625 disk.sda.out‚ 312.54296875! proc.memory.rssMB‚ 2873.421875 disk.dm-1.out‚ 312.54296875 disk.sda.in‚0 disk.nvme0n1.in‚ 1067.5703125" disk.nvme0c0n1.in‚ 1067.5703125) disk./.usagePercent‚73.87728331071729 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 1067.5703125$ memory_percent‚94.21517848968506$ disk./.usageGB‚1100.824909210205 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 312.54296875 proc.cpu.threads‚17 network.sent‚ 2.4956635e+07 disk.sda1.out‚0& disk.nvme0n1.out‚47191.54443359375 disk.sda3.in‚0# disk.dm-0.out‚47191.54443359375 cpu‚25.348089096119054( disk.nvme0c0n1.out‚47191.54443359375ÖïÈ£2™&- _wandbruntime‚7893  _runtime‚7893Çþ¢ Êš&Ä ' lr-AdamW/pg1‚1.2481978342544506e-05 ' lr-AdamW/pg2‚1.2481978342544506e-05 trainer/global_step‚10449 % _timestamp‚1.7884362091516898e+09 _runtime‚7893.856462381 _step‚856ÅŠÚÏ›&É trainer/global_step‚10449 % _timestamp‚1.7884362091516898e+09 _runtime‚7893.856462381 _step‚856 ' lr-AdamW/pg1‚1.2481978342544506e-05 ' lr-AdamW/pg2‚1.2481978342544506e-05ØJvc|2œ&- _wandbruntime‚7893  _runtime‚7893¼-.#è'&âO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚663360209646080 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚600270303844760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 197255024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚607468484959744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591551919561728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8718384283032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8750598624256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚591551902522368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151921022 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚607468502962688 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138563572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8750598624256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49924714 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600269632367644 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151921020 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151921020 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9178370950144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚7893.857407492 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8749942261760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚672538770257408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚616218427221504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚616219101586944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚616219101586944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9178536671232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10449 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚616218427221504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8717729845276 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884362091524146e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88639316 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197234986 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 349156006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8749942261760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197234986 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚663360233586176 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚607468502962688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138564032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88639318 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088åÜÀí'ž&çO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚616218427221504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8717729845276 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884362091524146e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88639316 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197234986 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 349156006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8749942261760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197234986 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚663360233586176 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚607468502962688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138564032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88639318 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚663360209646080 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚600270303844760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 197255024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚607468484959744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591551919561728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8718384283032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8750598624256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚591551902522368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151921022 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚607468502962688 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138563572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8750598624256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49924714 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600269632367644 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151921020 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151921020 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9178370950144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚7893.857407492 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8749942261760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚672538770257408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚616218427221504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚616219101586944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚616219101586944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9178536671232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10449 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Ù2ÚI2Ÿ&- _wandbruntime‚7894  _runtime‚7894Ó³Ž– &ìM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBȘ_\&DeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138576686 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88647484 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚858 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600326024410492 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8750761689600 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151935036 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349208542 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8719200718204 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349188476 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197253440 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138576232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49928748 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9179240917504 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151935038 % _timestamp‚1.7884362098505719e+09 trainer/global_step‚10449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8751418065408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349208542 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49929200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8751418065408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚616275599688192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚591606806652928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚607524856001536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8718546280240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349188476 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚672601282560512âSUe÷&¡&ñM PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138576686 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88647484 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚858 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600326024410492 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8750761689600 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151935036 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349208542 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8719200718204 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349188476 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197253440 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138576232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49928748 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9179240917504 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151935038 % _timestamp‚1.7884362098505719e+09 trainer/global_step‚10449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8751418065408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349208542 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49929200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8751418065408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚616275599688192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚591606806652928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚607524856001536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8718546280240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349188476 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚672601282560512 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2ÚJ¤'E2¢&- _wandbruntime‚7894  _runtime‚78943 a£&Û /loss/FBt3/detection_loss‚2.3810065746307374 0loss/FRt1/assignment_loss‚1.0926293234030406 2loss/SRqqt1/assignment_loss‚0.9941788417371836 2loss/SRqqt3/assignment_loss‚1.2060668084657553 1loss/SRqqt4/detection_loss‚1.5902118713168774 2loss/SRqqt2/assignment_loss‚1.4126998754569664 3loss/FBt2/assignment_loss‚0.0020600194301449745 1loss/SRqqt1/detection_loss‚0.9554680806556435 /loss/FBt2/detection_loss‚2.3887672821680703 trainer/global_step‚10449 _runtime‚7894.556945089 1loss/SRqqt2/detection_loss‚1.7377084926614221 /loss/FBt4/detection_loss‚2.2610539964267184 0loss/FRt2/assignment_loss‚2.1826948470539516 1loss/FBt3/assignment_loss‚0.11999933114275337 1loss/SRqqt3/detection_loss‚2.4347120920817056 %loss/total_loss‚38.02185727687986 0loss/FRt3/assignment_loss‚2.8710113804915856 epoch‚8 _step‚859 1loss/FBt1/assignment_loss‚0.09218686152173557 .loss/FRt2/detection_loss‚2.279267720712556 /loss/FRt4/assignment_loss‚3.155482018632548 0loss/FBt4/assignment_loss‚0.4134044685966468 /loss/FRt3/detection_loss‚2.8022806891079606 /loss/FBt1/detection_loss‚0.5763545625424156 % _timestamp‚1.7884362098523047e+09 1loss/SRqqt4/assignment_loss‚1.363609570094337 /loss/FRt1/detection_loss‚1.6454088137381606 .loss/FRt4/detection_loss‚2.063593754810948CÝxçf¤&à 1loss/FBt1/assignment_loss‚0.09218686152173557 .loss/FRt2/detection_loss‚2.279267720712556 /loss/FRt4/assignment_loss‚3.155482018632548 0loss/FBt4/assignment_loss‚0.4134044685966468 /loss/FRt3/detection_loss‚2.8022806891079606 /loss/FBt1/detection_loss‚0.5763545625424156 % _timestamp‚1.7884362098523047e+09 1loss/SRqqt4/assignment_loss‚1.363609570094337 /loss/FRt1/detection_loss‚1.6454088137381606 .loss/FRt4/detection_loss‚2.063593754810948 /loss/FBt3/detection_loss‚2.3810065746307374 0loss/FRt1/assignment_loss‚1.0926293234030406 2loss/SRqqt1/assignment_loss‚0.9941788417371836 2loss/SRqqt3/assignment_loss‚1.2060668084657553 1loss/SRqqt4/detection_loss‚1.5902118713168774 2loss/SRqqt2/assignment_loss‚1.4126998754569664 3loss/FBt2/assignment_loss‚0.0020600194301449745 1loss/SRqqt1/detection_loss‚0.9554680806556435 /loss/FBt2/detection_loss‚2.3887672821680703 trainer/global_step‚10449 _runtime‚7894.556945089 1loss/SRqqt2/detection_loss‚1.7377084926614221 /loss/FBt4/detection_loss‚2.2610539964267184 0loss/FRt2/assignment_loss‚2.1826948470539516 1loss/FBt3/assignment_loss‚0.11999933114275337 1loss/SRqqt3/detection_loss‚2.4347120920817056 %loss/total_loss‚38.02185727687986 0loss/FRt3/assignment_loss‚2.8710113804915856 epoch‚8 _step‚859ÛåD&U¥&‚(:’ ü½åÔ¥õ‚Ž gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.05' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.010000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãå¯ ¦&‚(:• ü½åÔ’Í®‘ disk.nvme0n1.in‚ 1067.5703125! disk.rbd2.out‚209.94677734375 disk.rbd3.in‚0 disk.dm-0.in‚ 1067.5703125 network.sent‚ 2.4961246e+07 disk.sda3.in‚0# disk.dm-0.out‚47290.72607421875 disk.dm-1.in‚0! proc.memory.rssMB‚ 2873.421875 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1100.8250160217285 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda1.out‚0 network.recv‚ 2.690844e+06" disk.nvme0c0n1.in‚ 1067.5703125( disk.nvme0c0n1.out‚47290.72607421875 disk.sda2.out‚0 disk.sda1.in‚0 proc.memory.availableMB‚948 cpu‚25.347545503312183) disk./.usagePercent‚73.87729047892824& disk.nvme0n1.out‚47290.72607421875 disk.rbd1.out‚ 3.986328125 disk.sda3.out‚ 313.015625 disk.sda.out‚ 313.015625 disk.dm-1.out‚ 313.015625* proc.memory.percent‚17.537975311279297 disk.drbd1777.out‚0 memory_percent‚ 94.2138671875c.¯É§&‚(:’ ‹¾åÔÙÜÜ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.48' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.495999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ® e#¨&‚(:˜ ‹¾åÔžºù disk.sda3.out‚313.625& disk.nvme0n1.out‚47291.38134765625 cpu‚25.346961685609013 network.sent‚ 2.5070298e+07( disk.nvme0c0n1.out‚47291.38134765625 disk.rbd3.out‚0 disk.sda.out‚313.625 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0# disk.dm-0.out‚47291.38134765625$ memory_percent‚94.21236515045166 disk.sda2.out‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2873.421875* proc.memory.percent‚17.537975311279297" disk.nvme0c0n1.in‚ 1067.5703125 network.recv‚ 2.697895e+06! disk.rbd2.out‚210.13427734375 disk.dm-0.in‚ 1067.5703125 disk.rbd1.out‚ 3.986328125 disk.nvme0n1.in‚ 1067.5703125 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚313.625 disk.sda3.in‚0$ disk./.usageGB‚1100.824722290039 disk.sda2.in‚0( proc.memory.availableMB‚ 948.24609375) disk./.usagePercent‚73.87727076634812™˜KÍ2©&- _wandbruntime‚7929  _runtime‚7929ž$¯—ɪ&à trainer/global_step‚10499 $ _timestamp‚1.788436244680911e+09 _runtime‚7929.385722455 _step‚860 ' lr-AdamW/pg1‚1.1746160648762335e-05 ' lr-AdamW/pg2‚1.1746160648762335e-05äù[àΫ&È ' lr-AdamW/pg1‚1.1746160648762335e-05 ' lr-AdamW/pg2‚1.1746160648762335e-05 trainer/global_step‚10499 $ _timestamp‚1.788436244680911e+09 _runtime‚7929.385722455 _step‚860Üà2¬&- _wandbruntime‚7929  _runtime‚7929V`¥^Ú­&âO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8791584430080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9221553665024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152621820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198161038 _step‚861 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198161038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚666442746529280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89047718 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚666442770469376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚594297109050368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚610286154905088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152621822 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198181076 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8759219622440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152621820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619077739335168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9221723580416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8790928067584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚675664494049792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89047716 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884362446816366e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 QCDeviceStatsMonitor.on_train_batØÒ"Ìch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619077064969728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 350782858 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚594297126089728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8758565184684 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 350802898 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139200597 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844L6ž í'®&çO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8791584430080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚7929.386511962 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198161038 _step‚861 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198161038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚666442746529280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89047718 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚666442770469376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚594297109050368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚610286154905088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152621822 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198181076 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8759219622440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152621820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619077739335168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9221723580416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8790928067584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚675664494049792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89047716 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884362446816366e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619077064969728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 350782858 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚594297126089728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8758565184684 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 350802898 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139200597 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8791584430080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9221553665024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152621820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89055886 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50157910 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884362453452344e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350835394 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198199556 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚619134911815168 _runtime‚7930.050457579 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚603112066277900 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚603111394800576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚619134237436416 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139213358 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50157474 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚594352030220288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152635836 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350835394 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚675727001248256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152635838 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚675726817891840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚594352013180928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚610342489940992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚666504397266944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198179492 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 trainer/global_step‚10499 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8792403871232 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8791747495424 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚610342507943936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚619134911815168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8791747495424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9222580041216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8759381619648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚610342507943936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8792403871232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89055886 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50157910 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884362453452344e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350835394 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198199556 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚619134911815168 _runtime‚7930.050457579 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚603112066277900 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚603111394800576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚619134237436416 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139213358 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50157474 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚594352030220288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152635836 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350835394 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚675727001248256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152635838 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚675726817891840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚594352013180928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚610342489940992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚666504397266944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198179492 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 trainer/global_step‚10499 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8792403871232 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8791747495424 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚610342507943936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚619134911815168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8791747495424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9222580041216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8759381619648 RBDeviceStatsMonitor.on_trainÝ49_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚610342507943936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0Þ¹+¹2²&- _wandbruntime‚7930  _runtime‚7930Ís¦[³&Õ 0loss/FBt3/assignment_loss‚0.4897930997956012 .loss/FRt2/detection_loss‚2.234268226302587 /loss/FBt1/detection_loss‚0.6180074092396135 2loss/SRqqt1/assignment_loss‚0.9059522075385049 .loss/FBt2/detection_loss‚3.113037198781967 /loss/FBt4/detection_loss‚1.8996090441942215 epoch‚8 /loss/FRt3/detection_loss‚2.7866834507269016 0loss/FRt3/assignment_loss‚3.1289071616004493 2loss/SRqqt3/assignment_loss‚1.7403959541999061 /loss/SRqqt1/detection_loss‚0.83772380098507 1loss/SRqqt2/detection_loss‚1.8574564396118631 2loss/FBt2/assignment_loss‚0.005989973325389049 %loss/total_loss‚40.42966756953936 0loss/FRt1/assignment_loss‚1.0387459775576224 0loss/FRt2/assignment_loss‚2.6409756449552684 1loss/FBt1/assignment_loss‚0.11198535902391292 0loss/SRqqt4/detection_loss‚1.823676417581737 _runtime‚7930.052022968 _step‚863 0loss/FRt4/assignment_loss‚2.9290639175309074 2loss/SRqqt2/assignment_loss‚1.6608761694206267 2loss/SRqqt4/assignment_loss‚1.6669996222311358 /loss/FRt1/detection_loss‚1.5287884934088931 .loss/FBt3/detection_loss‚2.260378454412733 % _timestamp‚1.7884362453473744e+09 0loss/FBt4/assignment_loss‚0.2530233546179564 .loss/FRt4/detection_loss‚2.388230229086346 0loss/SRqqt3/detection_loss‚2.509099963410147 trainer/global_step‚10499ºØE@`´&Ú 0loss/FRt1/assignment_loss‚1.0387459775576224 0loss/FRt2/assignment_loss‚2.6409756449552684 1loss/FBt1/assignment_loss‚0.11198535902391292 0loss/SRqqt4/detection_loss‚1.823676417581737 _runtime‚7930.052022968 _step‚863 0loss/FRt4/assignment_loss‚2.9290639175309074 2loss/SRqqt2/assignment_loss‚1.6608761694206267 2loss/SRqqt4/assignment_loss‚1.6669996222311358 /loss/FRt1/detection_loss‚1.5287884934088931 .loss/FBt3/detection_loss‚2.260378454412733 % _timestamp‚1.7884362453473744e+09 0loss/FBt4/assignment_loss‚0.2530233546179564 .loss/FRt4/detection_loss‚2.388230229086346 0loss/SRqqt3/detection_loss‚2.509099963410147 trainer/global_step‚10499 0loss/FBt3/assignment_loss‚0.4897930997956012 .loss/FRt2/detection_loss‚2.234268226302587 /loss/FBt1/detection_loss‚0.6180074092396135 2loss/SRqqt1/assignment_loss‚0.9059522075385049 .loss/FBt2/detection_loss‚3.113037198781967 /loss/FBt4/detection_loss‚1.8996090441942215 epoch‚8 /loss/FRt3/detection_loss‚2.7866834507269016 0loss/FRt3/assignment_loss‚3.1289071616004493 2loss/SRqqt3/assignment_loss‚1.7403959541999061 /loss/SRqqt1/detection_loss‚0.83772380098507 1loss/SRqqt2/detection_loss‚1.8574564396118631 2loss/FBt2/assignment_loss‚0.005989973325389049 %loss/total_loss‚40.42966756953936ßѷ蓵&‚(:ˆ š¾åÔôœ„Ž gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚139.512' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.9024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q!M^,¶&‚(:¡ š¾åÔÒÅ…‘ disk.sda2.in‚0 disk.rbd0.in‚0" disk./.usageGB‚1100.8251953125 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1067.5703125 disk.sda2.out‚0* proc.memory.percent‚17.537975311279297 disk.drbd1777.in‚0 disk.sda.in‚0 disk.rbd0.out‚0$ memory_percent‚94.21379566192627 proc.cpu.threads‚17 disk.rbd3.in‚0! proc.memory.rssMB‚ 2873.421875 network.recv‚ 2.701671e+06! disk.rbd2.out‚211.66162109375 disk.dm-1.in‚0& disk.nvme0n1.out‚47390.73291015625 disk.rbd1.in‚0# disk.dm-0.out‚47390.73291015625 disk.sda.out‚ 314.1484375 disk.sda1.in‚0 disk.nvme0n1.in‚ 1067.5703125" disk.nvme0c0n1.in‚ 1067.5703125( disk.nvme0c0n1.out‚47390.73291015625 cpu‚25.34639497372997 network.sent‚ 2.5075492e+07 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 3.986328125 disk.sda3.out‚ 314.1484375 disk.dm-1.out‚ 314.1484375( proc.memory.availableMB‚ 948.01171875) disk./.usagePercent‚73.87730251128235E£™òž·&‚(:“ ©¾åÔ÷¤ë gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.465' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.893000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦Tˆ5¸&‚(:ª ©¾åÔö’“ disk.sda1.in‚0 disk.dm-1.in‚0.5 disk.rbd1.out‚ 4.072265625( disk.nvme0c0n1.out‚47490.60107421875! proc.memory.rssMB‚ 2873.421875 network.recv‚ 2.708411e+06 disk.sda2.in‚0! disk.rbd2.out‚211.72412109375 disk.rbd3.out‚0) disk./.usagePercent‚73.87729227098099 disk.dm-0.in‚ 1067.5703125& disk.nvme0n1.out‚47490.60107421875 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0.5 disk.rbd3.in‚0 disk.sda.in‚0.5" disk.nvme0c0n1.in‚ 1067.5703125$ memory_percent‚94.21424865722656$ proc.memory.availableMB‚947.9375 cpu‚25.345854366378486 disk.nvme0n1.in‚ 1067.5703125* proc.memory.percent‚17.537975311279297% disk./.usageGB‚1100.8250427246094 disk.rbd1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 318.15234375 disk.rbd0.in‚0 disk.sda3.out‚ 318.15234375 disk.drbd1777.out‚0 network.sent‚ 2.5184394e+07# disk.dm-0.out‚47490.60107421875 disk.sda.out‚ 318.15234375Kÿë2¹&- _wandbruntime‚7964  _runtime‚7964ònæɺ&à ' lr-AdamW/pg1‚1.1031508999331099e-05 ' lr-AdamW/pg2‚1.1031508999331099e-05 trainer/global_step‚10549 $ _timestamp‚1.788436280197428e+09 _runtime‚7964.902200085 _step‚864ëÊ2λ&È _runtime‚7964.902200085 _step‚864 ' lr-AdamW/pg1‚1.1031508999331099e-05 ' lr-AdamW/pg2‚1.1031508999331099e-05 trainer/global_step‚10549 $ _timestamp‚1.788436280197428e+09à×f*42¼&- _wandbruntime‚7964  _runtime‚7964pÏ›Uç'½&áO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚621935702717952 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9264901932032 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚605842387579576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199087090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669525307352576 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139838836 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8800054961848 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚613103788844544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153322620 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚10549 _runtime‚ 7964.90315296 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 352409710 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚678790023817728 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚613103806847488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153322620 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚613103806847488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8832570235904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669525283412480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199107128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621935702717952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚678790209284608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8831913873408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89456118 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚597042315578368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153322622 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199087090 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8832570235904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 352429750 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚621936377083392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 % _timestamp‚1.7884362801982048e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153322622 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚613103788844544 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139838390 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597042332617728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8799400524092 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚678790023817728 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚613103806847488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153322620 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚613103806847488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8832570235904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669525283412480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199107128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621935702717952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚678790209284608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8831913873408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89456118 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚597042315578368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153322622 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199087090 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8832570235904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 352429750 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚621936377083392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 % _timestamp‚1.7884362801982048e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153322622 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚613103788844544 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139838390 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597042332617728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8799400524092 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9264901932032 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚605842387579576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199087090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669525307352576 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139838836 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8800054961848 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚613103788844544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153322620 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚10549áè7Õ’¿&‚(:‡ ¸¾åÔë·ê gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚88.424' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ʃ‘Ž8À&‚(:­ ¸¾åÔÃÿŠ disk.sda.out‚ 319.11328125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚94.21427249908447 disk.rbd0.in‚0% disk./.usageGB‚1100.8250770568848 disk.rbd3.in‚0 disk.rbd1.out‚ 4.072265625 disk.sda2.in‚0 disk.dm-1.in‚0.5( disk.nvme0c0n1.out‚47589.69091796875 disk.rbd1.in‚0 disk.sda3.out‚ 319.11328125 disk.drbd1777.in‚0 cpu‚25.345288478901388 proc.cpu.threads‚17 disk.rbd2.in‚0 network.sent‚ 2.5189855e+07 disk.dm-0.in‚ 1067.5703125# disk.dm-0.out‚47589.69091796875 disk.nvme0n1.in‚ 1067.5703125& disk.nvme0n1.out‚47589.69091796875" disk.nvme0c0n1.in‚ 1067.5703125( proc.memory.availableMB‚ 947.93359375 network.recv‚ 2.712417e+06 disk.sda2.out‚0! proc.memory.rssMB‚ 2873.421875* proc.memory.percent‚17.537975311279297 disk.sda.in‚0.5! disk.rbd2.out‚211.73974609375 disk.sda3.in‚0.5 disk.rbd3.out‚0 disk.dm-1.out‚ 319.11328125 disk.rbd0.out‚0 disk.drbd1777.out‚0( disk./.usagePercent‚73.8772945750488t€Û%2Á&- _wandbruntime‚7965  _runtime‚7965jl& Â&ìM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139851183 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8800871397020 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8800216959056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 352462246 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678852532539904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9265762000384 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batchÚ~äò_end/active.small_pool.allocated‚ 199125608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352462246 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 _step‚866 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚621992875184640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚613160141883392 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139851637 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153336638 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50387351 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597097219708928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199105544 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚613160141883392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚678852720090624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153336638 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89464286 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚621993549563392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8833389677056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50386899 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720‹Xu÷&Ã&ñM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352442180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚621993549563392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 352442180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139851183 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8800871397020 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8800216959056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 352462246 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678852532539904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9265762000384 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199125608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352462246 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 _step‚866 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚621992875184640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚613160141883392 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139851637 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153336638 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50387351 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597097219708928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199105544 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚613160141883392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚678852720090624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153336638 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89464286 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚621993549563392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8833389677056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50386899 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615921440786944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚608628429446984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚681915758963712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50611090 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 354036562 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89864516 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9307755298304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154023420 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615921440786944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8872899679232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154023422 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚672607820295680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624794340466176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9307914727936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚8000.809903582 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414,Í/í'Ð&çO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50611533 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8873556041728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚681915575593984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624795014831616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚599787539145728 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚615921458789888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚615921458789888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 354056602 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200013142 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154023422 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8872899679232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624794340466176 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140476051 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚869 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200013142 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615921440786944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚608628429446984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚681915758963712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50611090 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 354036562 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89864516 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9307755298304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154023420 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615921440786944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8872899679232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154023422 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚672607820295680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624794340466176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9307914727936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚8000.809903582 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8874375482880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50616139 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 354069032 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140488350 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8841052298464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624852187311616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8873719107072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚672669494973440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚599842426236928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚870 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚615977793825792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 354069032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200051660 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚681978269692928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200051660 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 354089098 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154037436 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608684150012716 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚599842443276288 trainer/global_step‚10599 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354089098 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140488825 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚672669471033344 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624851512932864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808wAVrõ&Ó&ïM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200051660 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 354089098 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154037436 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608684150012716 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚599842443276288 trainer/global_step‚10599 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354089098 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140488825 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚672669471033344 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624851512932864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8874375482880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8841706736428 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200031596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89872686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89872684 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200031596 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚624852187311616 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚681978075850752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9308604817408 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚608683478535392 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8874375482880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50616139 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 354069032 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140488350 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8841052298464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624852187311616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8873719107072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚672669494973440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚599842426236928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚870 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚615977793825792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 354069032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200051660 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚681978269692928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096æúñ2Ô&- _wandbruntime‚8001  _runtime‚8001˜gšOWÕ&Ñ /loss/FBt1/detection_loss‚0.6387496123592333 /loss/FBt3/detection_loss‚2.3333887855211892 .loss/FBt4/detection_loss‚2.662062478065491 0loss/FBt2/assignment_loss‚0.6252430230379105 1loss/FBt4/assignment_loss‚0.16166003154066858 /loss/SRqqt2/detection_loss‚2.05297280917334 _runtime‚ 8001.49778495 _step‚871 0loss/FRt1/assignment_loss‚1.2332769774872323 2loss/SRqqt4/assignment_loss‚1.1385012517138473 .loss/FRt3/detection_loss‚2.722799322297496 1loss/FBt3/assignment_loss‚0.12900702368157604 /loss/FRt2/detection_loss‚2.2315363583083334 1loss/SRqqt4/detection_loss‚1.8722279257369492 epoch‚8 trainer/global_step‚10599 $ _timestamp‚1.788436316793094e+09 /loss/FRt2/assignment_loss‚2.278472060194382 1loss/SRqqt1/detection_loss‚0.9352506570223915 2loss/SRqqt3/assignment_loss‚1.3620304099338896 .loss/FRt4/detection_loss‚2.068994988474952 1loss/SRqqt3/detection_loss‚2.2812012758325126 %loss/total_loss‚41.92848877260083 .loss/FRt1/detection_loss‚1.693247838249511 .loss/FRt4/assignment_loss‚3.10242760181427 2loss/SRqqt2/assignment_loss‚1.3668234683573246 1loss/FBt1/assignment_loss‚0.06856213322072059 .loss/FBt2/detection_loss‚4.646481156349182 0loss/FRt3/assignment_loss‚3.2926590904112785 2loss/SRqqt1/assignment_loss‚1.0309124938171488F€rÖ&Ö 0loss/FRt3/assignment_loss‚3.2926590904112785 2loss/SRqqt1/assignment_loss‚1.0309124938171488 /loss/FBt1/detection_loss‚0.6387496123592333 /loss/FBt3/detection_loss‚2.3333887855211892 .loss/FBt4/detection_loss‚2.662062478065491 0loss/FBt2/assignment_loss‚0.6252430230379105 1loss/FBt4/assignment_loss‚0.16166003154066858 /loss/SRqqt2/detection_loss‚2.05297280917334 _runtime‚ 8001.49778495 _step‚871 0loss/FRt1/assignment_loss‚1.2332769774872323 2loss/SRqqt4/assignment_loss‚1.1385012517138473 .loss/FRt3/detection_loss‚2.722799322297496 1loss/FBt3/assignment_loss˜ÚÕŠê‚0.12900702368157604 /loss/FRt2/detection_loss‚2.2315363583083334 1loss/SRqqt4/detection_loss‚1.8722279257369492 epoch‚8 trainer/global_step‚10599 $ _timestamp‚1.788436316793094e+09 /loss/FRt2/assignment_loss‚2.278472060194382 1loss/SRqqt1/detection_loss‚0.9352506570223915 2loss/SRqqt3/assignment_loss‚1.3620304099338896 .loss/FRt4/detection_loss‚2.068994988474952 1loss/SRqqt3/detection_loss‚2.2812012758325126 %loss/total_loss‚41.92848877260083 .loss/FRt1/detection_loss‚1.693247838249511 .loss/FRt4/assignment_loss‚3.10242760181427 2loss/SRqqt2/assignment_loss‚1.3668234683573246 1loss/FBt1/assignment_loss‚0.06856213322072059 .loss/FBt2/detection_loss‚4.646481156349182ç¬n\ƒž×&‚(:“ å¾åÔðµó gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.463' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.492599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>qÅ0Ø&‚(:¥ å¾åÔÌا disk.dm-1.out‚ 320.765625 disk.rbd1.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 1067.5703125 disk.rbd0.out‚0* proc.memory.percent‚17.537975311279297 network.sent‚ 2.5390567e+07 disk.sda.out‚ 320.765625& disk.nvme0n1.out‚47792.53759765625 disk.sda3.out‚ 320.765625 disk.sda.in‚0.5' proc.memory.availableMB‚ 947.3515625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.nvme0n1.in‚ 1067.5703125 disk.rbd0.in‚0 disk.rbd2.out‚ 213.478515625 cpu‚25.34363344965511# disk.dm-0.out‚47792.53759765625 disk.sda1.in‚0( disk.nvme0c0n1.out‚47792.53759765625 disk.sda2.in‚0 disk.sda3.in‚0.5 disk.drbd1777.in‚0$ memory_percent‚94.21782493591309! proc.memory.rssMB‚ 2873.421875 network.recv‚ 2.728804e+06 disk.rbd2.in‚0) disk./.usagePercent‚73.87756312695221 disk.rbd3.in‚0 disk.dm-0.in‚ 1067.5703125 disk.dm-1.in‚0.5 disk.rbd1.out‚ 4.1669921875 disk.sda2.out‚0% disk./.usageGB‚1100.8290786743164PÐÏžÙ&‚(:“ ô¾åÔÝí…Ž gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.271' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.454199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢¶¾]5Ú&‚(:ª ô¾åÔ§é‰ disk.nvme0n1.in‚ 1067.5703125 disk.rbd2.out‚ 213.494140625 disk.drbd1777.out‚0! proc.memory.rssMB‚ 2873.421875* proc.memory.percent‚17.537975311279297" disk.nvme0c0n1.in‚ 1067.5703125 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 947.59765625 proc.cpu.threads‚17( disk.nvme0c0n1.out‚47891.63525390625 disk.sda2.in‚0 disk.rbd2.in‚0 disk.sda.in‚0.5) disk./.usagePercent‚73.87756696706522% disk./.usageGB‚1100.8291358947754 disk.sda2.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 1067.5703125 disk.drbd1777.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚47891.63525390625 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 2.5498843e+07& disk.nvme0n1.out‚47891.63525390625 disk.sda3.in‚0.5 disk.dm-1.out‚ 321.40234375 cpu‚25.3430879089985 disk.rbd1.in‚0 disk.sda.out‚ 321.40234375 disk.dm-1.in‚0.5$ memory_percent‚94.21632289886475 network.recv‚ 2.73501e+06 disk.rbd1.out‚ 4.1669921875 disk.sda3.out‚ 321.40234375ñ´o2Û&- _wandbruntime‚8036  _runtime‚8036ªP>ÚÆÜ&À % lr-AdamW/pg2‚9.66629512084252e-06 trainer/global_step‚10649 % _timestamp‚1.7884363519831758e+09 _runtime‚8036.687927159 _step‚872 % lr-AdamW/pg1‚9.66629512084252e-06­8)ËÝ&Å % lr-AdamW/pg1‚9.66629512084252e-06 % lr-AdamW/pg2‚9.66629512084252e-06 trainer/global_step‚10649 % _timestamp‚1.7884363519831758e+09 _runtime‚8036.687927159 _step‚872èm-ÄO2Þ&- _wandbruntime‚8036  _runtime‚8036(=7‹è'ß&âO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚618739110732288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚627653652579840 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154724222 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9351087515648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8881725640664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200959232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚675690357178880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141112043 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8881071202908 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50839558 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685041283167744 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602532728634368 % _timestamp‚1.7884363519840357e+09 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141112476 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200939194 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90272916 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90272918 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200959232 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚618739092729344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8914541847552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9350925988864 trainer/global_step‚10649 _step‚873 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154724220 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚611414471314392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627652978214400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8913885485056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8913885485056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚611413799837276 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8914541847552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚627652978214400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚602532745673728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚618739110732288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ª´8 í'à&çO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627652978214400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8913885485056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8913885485056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚611413799837276 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8914541847552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚627652978214400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚602532745673728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚618739110732288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚618739110732288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚627653652579840 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154724222 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9351087515648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8881725640664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200959232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚675690357178880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141112043 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8881071202908 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50839558 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685041283167744 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602532728634368 % _timestamp‚1.7884363519840357e+09 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141112476 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200939194 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90272916 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90272918 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200959232 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚618739092729344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8914541847552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9350925988864 trainer/global_step‚10649 _step‚873 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154724220 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚611414471314392éžè˜”2á&- _wandbruntime‚8037  _runtime‚8037Ø”  â&ìM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8881887637872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90281086 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚618795445768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚618795463771136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884363526614523e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚627710150681088 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200957648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚675752007916544 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8882542075836 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8914704912896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚602587632764928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 355715950 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚611470191880124 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154738236 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10649 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90281084 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8915361288704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚685103986201088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9351954344448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9351788636672 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚611469520402800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141125141 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 355695884 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355715950 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚8037.366851474 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8915361288704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200977712 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154738238 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141124691 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚627710825059840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8914704912896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355695884 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50843607 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154738236 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10649 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90281084 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8915361288704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚685103986201088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9351954344448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9351788636672 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚611469520402800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141125141 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 355695884 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355715950 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚8037.366851474 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8915361288704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200977712 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154738238 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141124691 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚627710825059840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8914704912896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355695884 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50843607 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90281086 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚618795445768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚618795463771136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884363526614523e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚627710150681088 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200957648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚675752007916544 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8882542075836 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8914704912896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚602587632764928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 355715950êq3K2ä&- _wandbruntime‚8037  _runtime‚80374B¦2aå&Û /loss/FBt3/detection_loss‚2.5661659240722656 trainer/global_step‚10649 0loss/FRt2/assignment_loss‚2.4585999724192495 % _timestamp‚1.7884363526632056e+09 _runtime‚8037.368199474 .loss/FBt2/detection_loss‚2.968870908021927 %loss/total_loss‚39.19269309229436 0loss/FRt4/assignment_loss‚3.0243461164268286 2loss/SRqqt4/assignment_loss‚1.4947018475860965 /loss/FBt1/detection_loss‚0.4235784388696023 2loss/SRqqt1/assignment_loss‚0.9348105113163139 .loss/FRt1/detection_loss‚1.733350889912496 _step‚875 /loss/FRt4/detection_loss‚2.8464892635474333 1loss/SRqqt3/detection_loss‚1.9917434024256329 epoch‚8 2loss/SRqqt2/assignment_loss‚1.2821950306370855 1loss/SRqqt3/assignment_loss‚1.652865000935488 3loss/FBt2/assignment_loss‚0.0036099997159908526 /loss/FRt2/detection_loss‚2.4144534789598904 /loss/FRt3/detection_loss‚2.6079831016369357 1loss/SRqqt4/detection_loss‚1.8517976555289055 /loss/FBt4/detection_loss‚2.1920689259256636 0loss/FRt1/assignment_loss‚1.0869337717692058 1loss/FBt1/assignment_loss‚0.10764991264293784 1loss/FBt4/assignment_loss‚0.05045569834432432 0loss/SRqqt2/detection_loss‚1.720495694577694 /loss/FRt3/assignment_loss‚2.754530295347556 1loss/FBt3/assignment_loss‚0.26099820181261746 1loss/SRqqt1/detection_loss‚0.7639990498622259J?î:fæ&à .loss/FBt2/detection_loss‚2.968870908021927 %loss/total_loss‚39.19269309229436 0loss/FRt4/assignment_loss‚3.0243461164268286 2loss/SRqqt4/assignment_loss‚1.4947018475860965 /loss/FBt1/detection_loss‚0.4235784388696023 2loss/SRqqt1/assignment_loss‚0.9348105113163139 .loss/FRt1/detection_loss‚1.733350889912496 _step‚875 /loss/FRt4/detection_loss‚2.8464892635474333 1loss/SRqqt3/detection_loss‚1.9917434024256329 epoch‚8 2loss/SRqqt2/assignment_loss‚1.2821950306370855 1loss/SRqqt3/assignment_loss‚1.652865000935488 3loss/FBt2/assignment_loss‚0.0036099997159908526 /loss/FRt2/detection_loss‚2.4144534789598904 /loss/FRt3/detection_loss‚2.6079831016369357 1loss/SRqqt4/detection_loss‚1.8517976555289055 /loss/FBt4/detection_loss‚2.1920689259256636 0loss/FRt1/assignment_loss‚1.0869337717692058 1loss/FBt1/assignment_loss‚0.10764991264293784 1loss/FBt4/assignment_loss‚0.05045569834432432 0loss/SRqqt2/detection_loss‚1.720495694577694 /loss/FRt3/assignment_loss‚2.754530295347556 1loss/FBt3/assignment_loss‚0.26099820181261746 1loss/SRqqt1/detection_loss‚0.7639990498622259 /loss/FBt3/detection_loss‚2.5661659240722656 trainer/global_step‚10649 0loss/FRt2/assignment_loss‚2.4585999724192495 % _timestamp‚1.7884363526632056e+09 _runtime‚8037.368199474ë¥á,êžç&‚(:“ ƒ¿åÔ²‰é gpu.0.gpu‚21.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚145.032' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.006400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"­¦Û8è&‚(:­ ƒ¿åÔ„ÿµ disk.sda2.out‚0 disk.sda3.in‚0.5 disk.sda.in‚0.5 disk.dm-1.out‚ 322.07421875 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 322.07421875 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 1067.5703125 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 947.32421875% disk./.usageGB‚1100.8289642333984 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 1067.5703125 disk.dm-1.in‚0.5$ memory_percent‚94.21799182891846 proc.cpu.threads‚17) disk./.usagePercent‚73.87755544672618# disk.dm-0.out‚47892.65380859375 disk.sda.out‚ 322.07421875* proc.memory.percent‚17.537975311279297 network.sent‚ 2.5505071e+07 disk.sda2.in‚0 disk.nvme0n1.in‚ 1067.5703125& disk.nvme0n1.out‚47892.65380859375 disk.rbd1.out‚ 4.22998046875 cpu‚25.34254060080397 network.recv‚ 2.739385e+06 disk.rbd2.out‚ 213.619140625 disk.sda1.out‚0! proc.memory.rssMB‚ 2873.421875( disk.nvme0c0n1.out‚47892.65380859375o³ÿ§’é&‚(:‡ ’¿åÔ‡¥‡Ž gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚91.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇSõ3ê&‚(:¨ ’¿åÔÈÇ disk.rbd2.in‚0 disk.sda3.out‚ 322.6015625 disk.sda.out‚ 322.6015625 proc.cpu.threads‚17( disk./.usagePercent‚73.8775592868392# disk.dm-0.out‚47991.87158203125 disk.sda.in‚0.5 disk.dm-1.in‚0.5 disk.rbd0.out‚0 disk.dm-0.in‚ 1067.5703125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1067.5703125& disk.nvme0n1.out‚47991.87158203125( disk.nvme0c0n1.out‚47991.87158203125$ memory_percent‚94.21720504760742! proc.memory.rssMB‚ 2873.421875 network.sent‚ 2.5613043e+07 network.recv‚ 2.745014e+06* proc.memory.percent‚17.537975311279297 disk.sda3.in‚0.5 disk.drbd1777.out‚0& proc.memory.availableMB‚ 947.453125 cpu‚25.341996694911455 disk.rbd3.in‚0 disk.dm-1.out‚ 322.6015625 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 1067.5703125 disk.sda2.in‚0 disk.rbd2.out‚ 213.630859375 disk.drbd1777.in‚0% disk./.usageGB‚1100.8290214538574 disk.sda1.in‚0 disk.rbd1.out‚ 4.22998046875 disk.sda2.out‚0 disk.rbd0.in‚0ß‘Ó‘ë&‚(:† ¡¿åÔÚ€á gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø"zì&‚(:“ ¡¿åÔ›†ì disk.sda3.in‚0.5$ disk./.usageGB‚1100.829231262207 disk.dm-0.in‚ 1067.5703125 disk.sda.in‚0.5 disk.rbd0.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 946.99609375 proc.cpu.threads‚17 network.sent‚ 2.561938e+07 disk.sda.out‚323.25 disk.rbd1.in‚0 disk.drbd1777.in‚0 disk.sda3.out‚323.25) disk./.usagePercent‚73.87757336725358 disk.sda1.in‚0 disk.dm-1.out‚323.25 disk.nvme0n1.in‚ 1067.5703125$ disk.nvme0n1.out‚48091.361328125 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0! disk.rbd2.out‚213.71728515625 network.recv‚ 2.749807e+06! disk.dm-0.out‚48091.361328125" disk.nvme0c0n1.in‚ 1067.5703125 disk.sda2.out‚0 cpu‚25.341429488113295 disk.rbd2.in‚0* proc.memory.percent‚17.537975311279297 disk.sda1.out‚0 disk.rbd1.out‚ 4.29296875& disk.nvme0c0n1.out‚48091.361328125 disk.sda2.in‚0 disk.dm-1.in‚0.5$ memory_percent‚94.21999454498291! proc.memory.rssMB‚ 2873.421875 ÖùH2í&- _wandbruntime‚8072  _runtime‚8072¬ ›ºÆî&À _runtime‚8072.841200325 _step‚876 % lr-AdamW/pg1‚9.01601740031125e-06 % lr-AdamW/pg2‚9.01601740031125e-06 trainer/global_step‚10699 % _timestamp‚1.7884363881364267e+09Ç`]^Ëï&Å _runtime‚8072.841200325 _step‚876 % lr-AdamW/pg1‚9.01601740031125e-06 % lr-AdamW/pg2‚9.01601740031125e-06 trainer/global_step‚10699 % _timestamp‚1.7884363881364267e+09ìßþ ®2ð&- _wandbruntime‚8072  _runtime‚8072#5½ç'ñ&áO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚8072.842185872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚605277935162368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201885284 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201865246 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141749975 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚630511615962624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8921906542316 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8954871290880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚621556762674688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8955527653376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚678772918002176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155425020 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚621556762674688 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 357290266 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357310306 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90681316 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688167062288896 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201885284 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8922560980072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚630511615962624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚621556744671744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚678772894062080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚877 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51068659 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚630512290328064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9394144286720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201865246 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚621556744671744 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155425022 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚688166874724864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚08sJ7Ÿ ò&æO BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚678772918002176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155425020 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚621552 …EM6762674688 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 357290266 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357310306 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90681316 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688167062288896 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201885284 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8922560980072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚630511615962624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚621556744671744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚678772894062080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚877 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51068659 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚630512290328064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9394144286720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201865246 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚621556744671744 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155425022 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚688166874724864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚8072.842185872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚605277935162368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201885284 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201865246 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141749975 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚630511615962624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8921906542316 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8954871290880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚621556762674688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8955527653376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0í{‚óö2ó&- _wandbruntime‚8073  _runtime‚8073eþ‹êò&ô&ìM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚10699 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155439036 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357342802 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357342802 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8956347094528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚621613097710592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90689484 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8955690718720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8955690718720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 357322736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155439038 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51073618 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚630569462808064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201883700 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚605332839292928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51073173 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141763104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9395003273728 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201903764 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚678834544799744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8073.514878636 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚614256233747532 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357322736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚621613097710592 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141762657 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688229572013568 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚614255562270208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚630569462808064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚621613115713536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844,:î\÷&õ&ñM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚621613097710592 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141762657 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688229572013568 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚614255562270208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚630569462808064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚621613115713536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚10699 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155439036 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357342802 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357342802 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8956347094528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚621613097710592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90689484 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8955690718720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8955690718720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 357322736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155439038 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51073618 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚630569462808064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201883700 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚605332839292928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51073173 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141763104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9395003273728 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201903764 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚678834544799744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8073.514878636 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚614256233747532 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357322736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0îçX)2ö&- _wandbruntime‚8073  _runtime‚8073­—£+Y÷&Ó /loss/FRt3/assignment_loss‚3.507348461151123 2loss/SRqqt2/assignment_loss‚1.3998800382531922 /loss/FRt4/detection_loss‚2.3027800964191556 /loss/FBt3/detection_loss‚2.9338311354319253 /loss/FBt2/detection_loss‚2.6333610117435455 epoch‚8 /loss/FRt2/detection_loss‚2.1852970123291016 0loss/FBt1/assignment_loss‚0.0638117916924682 1loss/FBt3/assignment_loss‚0.20031949575862504 1loss/SRqqt2/detection_loss‚1.7356530777339279 /loss/FBt1/detection_loss‚0.5005969932837229 .loss/FBt4/detection_loss‚1.911915022134781 0loss/FRt4/assignment_loss‚3.1528266283373036 1loss/SRqqt4/assignment_loss‚1.546309525722807 0loss/FBt2/assignment_loss‚0.1459687552651303 0loss/FBt4/assignment_loss‚0.3355556199327111 1loss/SRqqt1/detection_loss‚0.7909292094934155 trainer/global_step‚10699 0loss/FRt1/assignment_loss‚1.5751248598098755 1loss/SRqqt3/assignment_loss‚1.718072123020082 %loss/total_loss‚41.65487201793869 _runtime‚8073.516024705 /loss/FRt3/detection_loss‚2.8170857930183413 1loss/SRqqt1/assignment_loss‚1.046634075402374 _step‚879 /loss/FRt1/detection_loss‚1.8940229800840218 0loss/SRqqt3/detection_loss‚2.285954642134744 1loss/SRqqt4/detection_loss‚1.8319152626124295 % _timestamp‚1.7884363888113568e+09 .loss/FRt2/assignment_loss‚3.13967840717389ßàC^ø&Ø epoch‚8 /loss/FRt2/detection_loss‚2.1852970123291016 0loss/FBt1/assignment_loss‚0.0638117916924682 1loss/FBt3/assignment_loss‚0.20031949575862504 1loss/SRqqt2/detection_loss‚1.7356530777339279 /loss/FBt1/detection_loss‚0.5005969932837229 .loss/FBt4/detection_loss‚1.911915022134781 0loss/FRt4/assignment_loss‚3.1528266283373036 1loss/SRqqt4/assignment_loss‚1.546309525722807 0loss/FBt2/assignment_loss‚0.1459687552651303 0loss/FBt4/assignment_loss‚0.3355556199327111 1loss/SRqqt1/detection_loss‚0.7909292094934155 trainer/global_step‚10699 0loss/FRt1/assignment_loss‚1.5751248598098755 1loss/SRqqt3/assignment_loss‚1.718072123020082 %loss/total_loss‚41.65487201793869 _runtime‚8073.516024705 /loss/FRt3/detection_loss‚2.8170857930183413 1loss/SRqqt1/assignment_loss‚1.046634075402374 _step‚879 /loss/FRt1/detection_loss‚1.8940229800840218 0loss/SRqqt3/detection_loss‚2.285954642134744 1loss/SRqqt4/detection_loss‚1.8319152626124295 % _timestamp‚1.7884363888113568e+09 .loss/FRt2/assignment_loss‚3.13967840717389 /loss/FRt3/assignment_loss‚3.507348461151123 2loss/SRqqt2/assignment_loss‚1.3998800382531922 /loss/FRt4/detection_loss‚2.3027800964191556 /loss/FBt3/detection_loss‚2.9338311354319253 /loss/FBt2/detection_loss‚2.6333610117435455ïç‰à’ù&‚(:‡ °¿åÔòÎŽ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚91.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚Ç5ú&‚(:ª °¿åÔž¨¬ disk.sda1.in‚0 disk.dm-1.in‚0.5& disk.nvme0n1.out‚48190.52490234375 proc.cpu.threads‚17! disk.rbd2.out‚213.73291015625 disk.sda3.in‚0.5 disk.sda.in‚0.5 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 323.84765625 network.recv‚ 2.75401e+06) disk./.usagePercent‚73.87757336725358( disk.nvme0c0n1.out‚48190.52490234375 disk.drbd1777.in‚0# disk.dm-0.out‚48190.52490234375$ memory_percent‚94.22030448913574! proc.memory.rssMB‚ 2873.421875 disk.rbd1.in‚0 network.sent‚ 2.5624848e+07" disk.nvme0c0n1.in‚ 1067.5703125 disk.rbd2.in‚0 cpu‚25.340918083141084$ disk./.usageGB‚1100.829231262207 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1067.5703125' proc.memory.availableMB‚ 946.9453125 disk.dm-1.out‚ 323.84765625 disk.rbd1.out‚ 4.29296875* proc.memory.percent‚17.537975311279297 disk.rbd0.out‚0 disk.dm-0.in‚ 1067.5703125 disk.sda.out‚ 323.84765625¹Î¬jû&‚(:’ ¿¿åÔÔ§Û gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.045999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èn Nþü&‚(:¦ ¿¿åÔôÌ„) disk./.usagePercent‚73.87757234322343" disk.nvme0c0n1.in‚ 1067.5703125 disk.sda3.in‚0.5 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 324.4140625# disk.dm-0.out‚48191.04052734375! proc.memory.rssMB‚ 2873.421875 proc.cpu.threads‚17 network.recv‚ 2.760699e+06 disk.dm-1.in‚0.5 disk.rbd1.in‚0 network.sent‚ 2.5733642e+07$ disk./.usageGB‚1100.829216003418 disk.dm-0.in‚ 1067.5703125 disk.sda.out‚ 324.4140625 disk.dǦD±3m-1.out‚ 324.4140625 disk.nvme0n1.in‚ 1067.5703125* proc.memory.percent‚17.537975311279297 disk.sda1.in‚0! disk.rbd2.out‚215.30712890625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda.in‚0.5& disk.nvme0n1.out‚48191.04052734375 disk.rbd1.out‚ 4.29296875& proc.memory.availableMB‚ 946.828125 cpu‚25.34035231914007( disk.nvme0c0n1.out‚48191.04052734375 disk.sda2.in‚0$ memory_percent‚94.22101974487305]ÊW2ý&- _wandbruntime‚8108  _runtime‚8108ó+IÝÆþ&À trainer/global_step‚10749 % _timestamp‚1.7884364240960467e+09 _runtime‚8108.800867534 _step‚880 % lr-AdamW/pg1‚8.38747468189185e-06 % lr-AdamW/pg2‚8.38747468189185e-06t\SóËÿ&Å % lr-AdamW/pg1‚8.38747468189185e-06 % lr-AdamW/pg2‚8.38747468189185e-06 trainer/global_step‚10749 % _timestamp‚1.7884364240960467e+09 _runtime‚8108.800867534 _step‚880ðº\êi2€'- _wandbruntime‚8108  _runtime‚8108ÿÀ:µç''áO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91089716 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156125822 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚624374414617088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358937158 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156125822 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202811336 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616985883572092 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚608023141690368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚624374396614144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9437115822592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633370928076288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788436424096932e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8995857096704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51298895 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142388611 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8996513459200 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202791298 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358917118 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚633370928076288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9436958490112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156125820 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚691292570707968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633370253710848 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91089718 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202811336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8995857096704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚608023158729728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8962741881724 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚691292389435392 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8963396319480 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142389068 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156125820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358937158 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156125822 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202811336 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616985883572092 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚608023141690368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚624374396614144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9437115822592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633370928076288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788436424096932e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8995857096704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51298895 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142388611 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8996513459200 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202791298 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358917118 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚633370928076288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9436958490112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156125820 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚691292570707968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633370253710848 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91089718 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202811336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8995857096704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚608023158729728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8962741881724 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚691292389435392 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8963396319480 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142389068 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156125820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8997332900352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8996676524544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚691354907480064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156139838 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚882 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156139838 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202829816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91097884 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156139836 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚624430767655936 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202809752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚633427426177536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358949588 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51303949 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9437983116288 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358969654 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚624430749652992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚633428100556288 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202829816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91097886 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358949588 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51303510 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8963558316688 % _timestamp‚1.7884364247759552e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142401394 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8964212754652 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚624430767655936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358969654 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚691355088739328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚681917081682944º®ˆBž…'îM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚617041604137616 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚624430767655936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358969654 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚691355088739328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitåìáyV"or.on_train_batch_end/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚681917081682944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚624430749652992 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142401835 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9437825797120 _runtime‚ 8109.481087 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8997332900352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8996676524544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚691354907480064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156139838 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚882 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156139838 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202829816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91097884 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156139836 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚624430767655936 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202809752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚633427426177536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358949588 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51303949 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9437983116288 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358969654 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚624430749652992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚633428100556288 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202829816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91097886 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358949588 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51303510 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8963558316688 % _timestamp‚1.7884364247759552e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142401394 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8964212754652 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚627192066559488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚619772596916616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156826620 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627192048556544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9036842902528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚636228891459072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684937967828480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚636229565824512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9003577221132 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884364599924178e+09 _runtime‚8144.697332912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360564010 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143027590 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚636228891459072 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚627192066559488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9037499265024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203737388 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91498118 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156826622 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚684937991768576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9004231658888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51529017 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9037499265024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚610768348218368 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203737388 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143027133 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚627192066559488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚619772596916616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156826620 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627192048556544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9036842902528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚636228891459072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684937967828480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚636229565824512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9003577221132 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884364599924178e+09 _runtime‚8144.697332912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360564010 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143027590 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚636228891459072 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚627192066559488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9037499265024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203737388 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91498118 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allRþ´ ocation.large_pool.allocated‚ 156826622 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚684937991768576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9004231658888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51529017 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9037499265024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚610768348218368 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203737388 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143027133 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360596506 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51534012 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚636286063925760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9038318706176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚694480590012416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156840636 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360596506 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 172870656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚694480762883072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9037662330368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884364606703584e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9037662330368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143039851 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚10799 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚636286738304512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627248401595392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚886 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚636286738304512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360576440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203755868 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51533567 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚619828317482348 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156840636 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627248401595392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚8145.375552911 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚610823252348928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚627248419598336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627248419598336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203735804 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203735804 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91506284 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414ÔNW÷&—'ñM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚694480762883072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9037662330368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884364606703584e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9037662330368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143039851 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚10799 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚636286738304512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627248401595392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚886 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚636286738304512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360576440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203755868 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51533567 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚619828317482348 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156840636 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627248401595392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚8145.375552911 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚610823252348928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚627248419598336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627248419598336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203735804 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203735804 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91506284 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360596506 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51534012 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚636286063925760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9038318706176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚694480590012416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156840636 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360596506 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 172870656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0övVÇî2˜'- _wandbruntime‚8145  _runtime‚8145S+Ox[™'Õ /loss/FRt2/assignment_loss‚2.635065173560923 2loss/SRqqt1/assignment_loss‚0.8212384100504485 /loss/FBt1/detection_loss‚0.5844500358957954 /loss/FBt2/detection_loss‚2.1899220824241636 1loss/SRqqt4/assignment_loss‚1.426492070968626 1loss/FBt1/assignment_loss‚0.06644048453542951 1loss/SRqqt2/detection_loss‚1.6286339044570923 0loss/FRt3/assignment_loss‚2.8298716247081757 0loss/FRt4/assignment_loss‚2.7904230501916674 _step‚887 &loss/total_loss‚37.721672241369376 epoch‚8 /loss/FBt2/assignment_loss‚0.324225456728891 0loss/FBt3/assignment_loss‚0.2836331800291581 /loss/FBt4/detection_loss‚3.1700064837932587 /loss/FBt3/detection_loss‚1.9149258817945207 % _timestamp‚1.7884364606721437e+09 /loss/FRt2/detection_loss‚2.2923658730631526 2loss/SRqqt2/assignment_loss‚1.1204555200887485 -loss/FRt3/detection_loss‚3.02364394813776 1loss/SRqqt4/detection_loss‚1.6486630446072352 0loss/FRt1/assignment_loss‚0.7405662142804691 .loss/FRt4/detection_loss‚2.269431515534719 _runtime‚8145.376815771 trainer/global_step‚10799 2loss/SRqqt3/assignment_loss‚1.4707909304733304 0loss/FBt4/assignment_loss‚0.1956036516930908 /loss/FRt1/detection_loss‚1.4463784441884076 1loss/SRqqt1/detection_loss‚0.7522538308852486 0loss/SRqqt3/detection_loss‚2.096191429279067þí›n`š'Ú _step‚887 &loss/total_loss‚37.721672241369376 epoch‚8 /loss/FBt2/assignment_loss‚0.324225456728891 0loss/FBt3/assignment_loss‚0.2836331800291581 /loss/FBt4/detection_loss‚3.1700064837932587 /loss/FBt3/detection_loss‚1.9149258817945207 % _timestamp‚1.7884364606721437e+09 /loss/FRt2/detection_loss‚2.2923658730631526 2loss/SRqqt2/assignment_loss‚1.1204555200887485 -loss/FRt3/detection_loss‚3.02364394813776 1loss/SRqqt4/detection_loss‚1.6486630446072352 0loss/FRt1/assignment_loss‚0.7405662142804691 .loss/FRt4/detection_loss‚2.269431515534719 _runtime‚8145.376815771 trainer/global_step‚10799 2loss/SRqqt3/assignment_loss‚1.4707909304733304 0loss/FBt4/assignment_loss‚0.1956036516930908 /loss/FRt1/detection_loss‚1.4463784441884076 1loss/SRqqt1/detection_loss‚0.7522538308852486 0loss/SRqqt3/detection_loss‚2.096191429279067 /loss/FRt2/assignment_loss‚2.635065173560923 2loss/SRqqt1/assignment_loss‚0.8212384100504485 /loss/FBt1/detection_loss‚0.5844500358957954 /loss/FBt2/detection_loss‚2.1899220824241636 1loss/SRqqt4/assignment_loss‚1.426492070968626 1loss/FBt1/assignment_loss‚0.06644048453542951 1loss/SRqqt2/detection_loss‚1.6286339044570923 0loss/FRt3/assignment_loss‚2.8298716247081757 0loss/FRt4/assignment_loss‚2.7904230501916674÷[Ú€ ž›'‚(:“ û¿åÔþÈÜ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.001' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.600199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü(Ë0œ'‚(:¥ û¿åÔ¹£À network.sent‚ 2.5951028e+07 disk.sda1.out‚0 cpu‚25.338221218947776 proc.cpu.threads‚17 disk.sda3.in‚0.5 disk.rbd3.out‚0 disk.dm-0.in‚ 1067.5703125$ disk.nvme0n1.out‚48489.271484375 disk.rbd2.out‚215.5498046875 disk.sda.out‚ 327.03515625% disk./.usageGB‚1100.8287506103516 disk.dm-1.out‚ 327.03515625 disk.nvme0n1.in‚ 1067.5703125 disk.rbd1.in‚0! proc.memory.rssMB‚ 2873.421875 disk.sda2.in‚0( proc.memory.availableMB‚ 947.92578125 network.recv‚ 2.781523e+06) disk./.usagePercent‚73.87754111030426 disk.sda3.out‚ 327.03515625 disk.rbd1.out‚ 4.29296875" disk.nvme0c0n1.in‚ 1067.5703125# memory_percent‚94.2143201828003* proc.memory.percent‚17.537975311279297 disk.sda.in‚0.5 disk.sda1.in‚0& disk.nvme0c0n1.out‚48489.271484375! disk.dm-0.out‚48489.271484375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.5 disk.rbd0.in‚0 disk.rbd2.in‚0€w­’'‚(:‡ ŠÀåÔÄðù gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.345 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷7Ã1ž'‚(:¦ ŠÀåÔŸù¡ disk.sda2.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚48588.353515625 disk.sda2.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1067.5703125 disk.sda1.out‚0 disk.dm-1.out‚ 327.69921875 disk.nvme0n1.in‚ 1067.5703125! proc.memory.rssMB‚ 2873.421875) disk./.usagePercent‚73.87754367037961 disk.rbd2.in‚0 disk.drbd1777.in‚0( proc.memory.availableMB‚ 948.00390625 disk.sda3.out‚ 327.69921875 disk.rbd3.in‚0! disk.dm-0.out‚48588.353515625 disk.rbd1.in‚0 disk.rbd1.out‚ 4.29296875* proc.memory.percent‚17.537975311279297 cpu‚25.337695513310873% disk./.usageGB‚1100.8287887573242" disk.nvme0c0n1.in‚ 1067.5703125 disk.sda3.in‚0.5 disk.sda.out‚ 327.69921875 disk.sda1.in‚0 network.sent‚ 2.6048172e+07 network.recv‚ 2.786954e+06 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda.in‚0.5$ disk.nvme0n1.out‚48588.353515625$ memory_percent‚94.21384334564209 disk.rbd2.out‚217.1396484375 disk.dm-1.in‚0.5Üñó42Ÿ'- _wandbruntime‚8180  _runtime‚8180¡M^È ' _step‚888 & lr-AdamW/pg1‚7.196113643990924e-06 & lr-AdamW/pg2‚7.196113643990924e-06 trainer/global_step‚10849 % _timestamp‚1.7884364954217033e+09 _runtime‚8180.126451184b?eÍ¡'Ç trainer/global_step‚10849 % _timestamp‚1.7884364954217033e+09 _runtime‚8180.126451184 _step‚888 & lr-AdamW/pg1‚7.196113643990924e-06 & lr-AdamW/pg2‚7.196113643990924e-06ø˜g82¢'- _wandbruntime‚8180  _runtime‚8180&6ûE"£'àO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51758507 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157527422 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157527420 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9044412560540 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚622557967306908 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9078485070848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚630009700498944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚639088203572736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143665023 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚630009718501888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 204643402 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91906516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚688020504711680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚613513571785728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚613513554746368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚10849 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204663440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91906518 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚697543998336512 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204663440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 $ _timestamp‚1.788436495422475e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚889 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630009718501888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 204643402 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9045066998296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚622558638784024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚639088203572736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639087529207296 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143665479 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚697543817063936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362170822 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9077828708352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚639087529207296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚ 8180.12744155 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚630009700498944@åª ë'¤'åO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9045066998296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚622558638784024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚639088203572736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639087529207296 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143665479 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚697543817063936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362170822 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9077828708352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚639087529207296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚ 8180.12744155 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚630009700498944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51758507 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157527422 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157527420 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9044412560540 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚622557967306908 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9078485070848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚630009700498944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚639088203572736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143665023 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚630009718501888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 204643402 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91906516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚688020504711680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚613513571785728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚613513554746368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚10849 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204663440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91906518 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚697543998336512 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204663440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 $ _timestamp‚1.788436495422475e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚889 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630009718501888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 204643402 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480ùc¦d"2¥'- _wandbruntime‚8180  _runtime‚8180Ä.nò&¦'ìM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9524185645568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362223358 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9524340867584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚613568475916288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚697606520257024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51763421 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884364960812056e+09 _step‚890 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157541436 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚639145376052736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚630066053537792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362203292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91914686 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362203292 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143677656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157541438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362223358 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚613568458876928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚630066053537792 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9079304512000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157541438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204681920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚697606341094912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9079304512000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143678107 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 204681920 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639144701673984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9045883433468 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204661856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639145376052736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9045228995504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204661856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630066071540736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9078648136192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51762972 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚622613687872432 _runtime‚8180.786458695 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9078648136192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91914684 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚622614359349756 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚630066071540736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚688082155449344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157541436 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430©Ži‘ §'ñM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630066071540736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9078648136192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51762972 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚622613687872432 _runtime‚8180.786458695 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9078648136192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91914684 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚622614359349756 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚630066071540736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚688082155449344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157541436 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9524185645568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362223358 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9524340867584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚613568475916288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚697606520257024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51763421 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884364960812056e+09 _step‚890 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157541436 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EîíatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚639145376052736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚630066053537792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362203292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91914686 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362203292 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143677656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157541438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362223358 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚613568458876928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚630066053537792 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9079304512000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157541438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204681920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚697606341094912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9079304512000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143678107 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 204681920 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639144701673984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9045883433468 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204661856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639145376052736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9045228995504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204661856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414úáR(\2¨'- _wandbruntime‚8180  _runtime‚8180ÙÐ4€\©'Ö 0loss/FRt1/assignment_loss‚1.0534855212484087 /loss/FRt3/assignment_loss‚3.451452171191191 1loss/FBt2/assignment_loss‚0.11162543227692368 0loss/SRqqt4/detection_loss‚2.365673255514015 %loss/total_loss‚41.35224686021887 trainer/global_step‚10849 % _timestamp‚1.7884364960830576e+09 /loss/FRt4/assignment_loss‚3.668294112319532 2loss/SRqqt1/assignment_loss‚0.7817124236999007 1loss/SRqqt4/assignment_loss‚1.565661971316193 0loss/FBt1/assignment_loss‚0.2850583229995067 /loss/FRt4/detection_loss‚2.0681556800138696 1loss/SRqqt2/detection_loss‚1.6328420218021151 /loss/FBt2/detection_loss‚2.4889742244373667 _step‚891 2loss/SRqqt2/assignment_loss‚1.1715457265575726 1loss/SRqqt1/detection_loss‚0.9688593314785976 0loss/FBt1/detection_loss‚0.49251852687889686 /loss/FBt4/detection_loss‚2.2666029930114746 0loss/FRt2/assignment_loss‚2.7914063842570194 1loss/SRqqt3/assignment_loss‚1.958485675630746 0loss/FBt4/assignment_loss‚0.3200133171242972 /loss/FRt2/detection_loss‚2.0874592358948756 .loss/FRt3/detection_loss‚2.868612928268237 epoch‚8 _runtime‚8180.788019044 1loss/FBt3/assignment_loss‚0.15961266539736904 /loss/FRt1/detection_loss‚1.8608664737216063 0loss/SRqqt3/detection_loss‚2.513343349651054 /loss/FBt3/detection_loss‚2.4199851155281067ïãÁaª'Û _step‚891 2loss/SRqqt2/assignment_loss‚1.1715457265575726 1loss/SRqqt1/detection_loss‚0.9688593314785976 0loss/FBt1/detection_loss‚0.49251852687889686 /loss/FBt4/detection_loss‚2.2666029930114746 0loss/FRt2/assignment_loss‚2.7914063842570194 1loss/SRqqt3/assignment_loss‚1.958485675630746 0loss/FBt4/assignment_loss‚0.3200133171242972 /loss/FRt2/detection_loss‚2.0874592358948756 .loss/FRt3/detection_loss‚2.868612928268237 epoch‚8 _runtime‚8180.788019044 1loss/FBt3/assignment_loss‚0.15961266539736904 /loss/FRt1/detection_loss‚1.8608664737216063 0loss/SRqqt3/detection_loss‚2.513343349651054 /loss/FBt3/detection_loss‚2.4199851155281067 0loss/FRt1/assignment_loss‚1.0534855212484087 /loss/FRt3/assignment_loss‚3.451452171191191 1loss/FBt2/assignment_loss‚0.11162543227692368 0loss/SRqqt4/detection_loss‚2.365673255514015 %loss/total_loss‚41.35224686021887 trainer/global_step‚10849 % _timestamp‚1.7884364960830576e+09 /loss/FRt4/assignment_loss‚3.668294112319532 2loss/SRqqt1/assignment_loss‚0.7817124236999007 1loss/SRqqt4/assignment_loss‚1.565661971316193 0loss/FBt1/assignment_loss‚0.2850583229995067 /loss/FRt4/detection_loss‚2.0681556800138696 1loss/SRqqt2/detection_loss‚1.6328420218021151 /loss/FBt2/detection_loss‚2.4889742244373667û·-ž«'‚(:“ ™ÀåÔ«ëæ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.564' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.912799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ôÍ­5¬'‚(:ª ™ÀåÔËüú) disk./.usagePercent‚73.87754443840221 disk.rbd0.in‚0 disk.drbd1777.out‚0$ memory_percent‚94.21553611755371 cpu‚25.337135331354155 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0.5 disk.sda.in‚0.5( disk.nvme0c0n1.out‚48688.18310546875 network.sent‚ 2.6054506e+07$ disk./.usageGB‚1100.828800201416 disk.dm-1.out‚ 328.47265625 disk.nvme0n1.in‚ 1067.5703125 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1067.5703125 disk.rbd1.out‚ 4.29296875! proc.memory.rssMB‚ 2873.421875 network.recv‚ 2.791442e+06" disk.nvme0c0n1.in‚ 1067.5703125 disk.rbd2.out‚217.2646484375 disk.drbd1777.in‚0 disk.dm-1.in‚0.5 disk.sda3.out‚ 328.47265625 disk.sda1.out‚0 disk.sda2.out‚0# disk.dm-0.out‚48688.18310546875 disk.sda.out‚ 328.47265625' proc.memory.availableMB‚ 947.7265625* proc.memory.percent‚17.537975311279297& disk.nvme0n1.out‚48688.18310546875 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0¶Vv’­'‚(:‡ ¨ÀåÔƲë gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚166.614' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.3228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ɖ!4®'‚(:© ¨ÀåÔ箓‘ network.sent‚ 2.6162507e+07) disk./.usagePercent‚73.87719242804263 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 1070.2265625 disk.drbd1777.in‚0& proc.memory.availableMB‚ 947.578125 disk.dm-1.out‚ 329.62890625 disk.sda3.in‚0.5! proc.memory.rssMB‚ 2873.421875* proc.memory.percent‚17.537975311279297 disk.dm-0.in‚ 1070.2265625 disk.nvme0n1.in‚ 1070.2265625 disk.rbd1.out‚ 4.29296875 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚0.5 disk.sda1.out‚0 disk.drbd1777.out‚0% disk./.usageGB‚1100.8235549926758 disk.dm-1.in‚0.5( disk.nvme0c0n1.out‚48789.65966796875 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 329.62890625 disk.sda1.in‚0 disk.sda3.out‚ 329.62890625# memory_percent‚94.2164421081543 cpu‚25.337795477950014 network.recv‚ 2.797455e+06# disk.dm-0.out‚48789.65966796875 disk.rbd2.out‚217.2802734375 proc.cpu.threads‚17& disk.nvme0n1.out‚48789.65966796875ˆ ‡¯'‚(:’ ·ÀåÔùÅÔ gpu.0.gpu‚18.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚87.061' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.412200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­éa9°'‚(:® ·ÀåÔï¬ disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0$ memory_percent‚94.21107769012451( proc.memory.availableMB‚ 948.45703125) disk./.usagePercent‚73.87720958054742 disk.nvme0n1.in‚ 1070.3515625" disk.nvme0c0n1.in‚ 1070.3515625 disk.rbd0.in‚0 disk.sda.out‚ 330.32421875 disk.rbd1.out‚ 4.29296875 disk.sda2.out‚0 disk.sda.in‚0.5 disk.rbd1.in‚0 disk.rbd3.out‚0) proc.memory.percent‚17.53408908843994 network.recv‚ 2.801773e+06 disk.sda1.out‚0 disk.dm-1.in‚0.5# proc.memory.rssMB‚ 2872.78515625 cpu‚25.346311771874788 proc.cpu.threads‚17% disk./.usageGB‚1100.8238105773926 disk.sda3.out‚ 330.32421875 disk.drbd1777.in‚0 network.sent‚ 2.6168744e+07& disk.nvme0n1.out‚48892.25048828125! disk.rbd2.out‚217.37060546875 disk.dm-0.in‚ 1070.3515625# disk.dm-0.out‚48892.25048828125 disk.dm-1.out‚ 330.32421875 disk.rbd0.out‚0 disk.sda3.in‚0.5 disk.sda1.in‚0( disk.nvme0c0n1.out‚48892.25048828125ƒW6Z“±'‚(:ˆ ÆÀåÔ¦¡Ü gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚165.784' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OlI\,²'‚(:¡ ÆÀåÔàÓã* proc.memory.percent‚17.531442642211914 cpu‚25.35428116793601 disk.rbd1.out‚ 4.29296875 disk.dm-0.in‚ 1071.15625 disk.sda1.out‚0& proc.memory.availableMB‚ 948.890625 disk.sda2.out‚0& disk.nvme0n1.out‚48892.88232421875 disk.nvme0c0n1.in‚ 1071.15625( disk.nvme0c0n1.out‚48892.88232421875 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda3.in‚0.5% disk./.usageGB‚1100.8243103027344 disk.dm-1.out‚ 331.1796875 disk.nvme0n1.in‚ 1071.15625! disk.rbd2.out‚217.38623046875 disk.drbd1777.in‚0 disk.sda.in‚0.5 network.recv‚ 2.804858e+06 disk.dm-1.in‚0.5 disk.sda.out‚ 331.1796875 disk.rbd3.out‚0 disk.sda3.out‚ 331.1796875 disk.drbd1777.out‚0 disk.rbd3.in‚0$ memory_percent‚94.20843124389648 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚48892.88232421875 disk.sda2.in‚0 network.sent‚ 2.6173066e+07( disk./.usagePercent‚73.8772431175344 disk.rbd0.out‚0" proc.memory.rssMB‚ 2872.3515625~ 2³'- _wandbruntime‚8249  _runtime‚8249úrnO*´'ÉT VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚616999898178560 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 9122618917376 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 364101956 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 158505428 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 144613522 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 9569067843072 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚626088929947188 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 _step‚892 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚642739795240448 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚642740306073600 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 158505428 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 144613522 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 9569067843072 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚626088929947188 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 _step‚892 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚642739795240448 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚642740306073600 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚158 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9089395378036 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 158516234 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚633654517185024 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚701456571901952 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9569317625856 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 158516234 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9088904463644 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 364120064 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚626124587316084 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚617035174898688 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚626124079362332 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚10880 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚92493092 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚642777501229056 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚633654499182080 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 144624673 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115343872 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9122491213824 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 9569432969728 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884365662053568e+09 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 9122491213824 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚158 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9089395378036 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 158516234 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚633654517185024 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚701456571901952 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9569317625856 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 158516234 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9088904463644 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 364120064 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚626124587316084 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚617035174898688 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚626124079362332 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚10880 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚92493092ý¿×ÜÎ2»'- _wandbruntime‚8263  _runtime‚8263á²µ,œ¼'·¥ :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26519501209259033 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01744186133146286 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5154428482055664 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.23425929248332977 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03099624067544937 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9783282279968262 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4340278208255768 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5230268836021423 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2361111044883728 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5646774768829346 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4915153682231903 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006163251120597124 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚41.52708907171866 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7071903347969055 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4289272129535675 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5075371265411377 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5627439022064209 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 +jet/accuracy_2_of_2‚0.11454810947179794 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.529420018196106 particle/accuracy_4_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9901536107063293 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08482784032821655 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5422271490097046 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 trainer/global_step‚10880 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23648647964000702 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 *jet/accuracy_1_of_3‚0.8782657980918884 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5902392864227295 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 +jet/accuracy_3_of_4‚0.07361501455307007 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26436781883239746 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07758620381355286 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_2_of_7‚0.5 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333432674408 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18394441902637482 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10640041530132294 jet/accuracy_3_of_7‚0 particle/accuracy_3_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10576117038726807 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9913495779037476 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10809538513422012 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_1_of_2‚0.773730993270874 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt4SRqqt0FBt/event_purIQOÀ"yity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121212482452393 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4614379107952118 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998635470867157 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34702378511428833 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3857503831386566 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1247267797589302 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18787887692451477 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.029428070411086082 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_1_of_1‚0.499502956867218 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.992340087890625 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23950017988681793 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.190476194024086 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20940664410591125 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6111111044883728 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5308223366737366 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02099432982504368 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0478997640311718 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4197531044483185 *jet/accuracy_3_of_6‚0.1538461595773697 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36069172620773315 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.37057727575302124 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18738901615142822 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983899593353271 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9766522645950317 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1688069850206375 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2984972298145294 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1739766001701355 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15111111104488373 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_6‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21483851969242096 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.641080379486084 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.181843563914299 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878621697425842 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.529420018196106 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.181843563914299 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5881928205490112 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.019736841320991516 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_2_of_3‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16540473699569702 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3236176669597626 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4319688081741333 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9272766709327698 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9923369288444519 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5340144038200378 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.203125 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 jet/accuracy_7_of_7‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19997796416282654 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9848790168762207 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 _step‚894 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.44780099391937256 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6075549721717834 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5199999809265137 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.43051478266716003 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1800329089164734 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.561073362827301 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_4_of_6‚1 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07372057437896729 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.007547169923782349 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5023846626281738 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20557819306850433 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4780701994895935 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 _runtime‚8263.213979561 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03067384660243988 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9858967065811157 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42013075947761536 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0052910055965185165 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5515188574790955 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.026171045377850533 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15789473056793213 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48793351650238037 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9686188101768494 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5275380611419678 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5880952477455139 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5022637248039246 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4958333671092987 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 particle/accuracy_1_of_4‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17038963735103607 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23141691088676453 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1358024626970291 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011944783851504326 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09196428209543228 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.431081086397171 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13619349896907806 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_6‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3742063641548157 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14861109852790833 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15494228899478912 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17696425318717957 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.365727037191391 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4700262248516083 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22991742193698883 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5266990661621094 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7validation_average_jet_accuracy‚0.44852787256240845 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5512820482254028 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5163424015045166 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5583860278129578 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2748015820980072 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5545635223388672 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.552556037902832 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_5‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3185184895992279 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23563113808631897 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.314353346824646 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04158986359834671 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8516348004341125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9986394643783569 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.44963526725769043 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5102732181549072 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6754385828971863 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5359246134757996 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013197427615523338 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5447530746459961 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137115478515625 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2870831787586212 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5315065979957581 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5621899962425232 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5455687642097473 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.037176571786403656 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4127906858921051 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1807130128145218 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5445402264595032 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3697747588157654 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2119491994380951 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18345864117145538 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018260840326547623 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14367815852165222 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5604895949363708 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1860005259513855 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878787517547607 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_7‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 particle/accuracy_1_of_5‚1 +particle/specificity‚0.9510868191719055 jet/accuracy_1_of_6‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9895887970924377 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.44685104489326477 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3611111044883728 particle/accuracy_2_of_6‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5034080743789673 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27288973331451416 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5185185074806213 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1595744639635086 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27776098251342773 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5430377721786499 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_6‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27097123861312866 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_7‚1 particle/accuracy_7_of_7‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895178079605103 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.42711541056632996 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_3_of_4‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18399354815483093 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9926888942718506 +particle/sensitivity‚0.3141602575778961 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5253510475158691 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 jet/accuracy_1_of_7‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08199996501207352 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40334826707839966 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6062595844268799 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20747126638889313 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 'particle/f_score‚0.3978305459022522 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22453705966472626 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.53570955991745 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4315440356731415 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.20631510019302368 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9871737957000732 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_6‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09056513011455536 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5178834795951843 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6204860806465149 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.33712121844291687 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.43703702092170715 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.03846153989434242 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16854242980480194 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 epoch‚8 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9909262657165527 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28169742226600647 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166567325592 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1953963190317154 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 particle/accuracy_1_of_3‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 % _timestamp‚1.7884365785068126e+09 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2589128315448761 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3965201675891876 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5813784599304199 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4435223937034607 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.008064515888690948 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5500505566596985 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 5 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.52923184633255 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17142947018146515 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5857216119766235 ( Purity/1FRt3SRqqt1FBt/FRt_purity‚0.5 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 particle/accuracy_5_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27955180406570435 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4583333432674408 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5453793406486511 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17642934620380402 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2153647243976593 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39791667461395264 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.005397115834057331 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.49928638339042664 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5970278978347778 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1640309989452362 +jet/accuracy_2_of_3‚0.26016706228256226 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.44869279861450195 particle/accuracy_1_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.540717363357544 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8178004026412964 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5731286406517029 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333432674408 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9865562319755554ƒ|âн'¼¥ *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22991742193698883 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5266990661621094 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7validation_average_jet_accuracy‚0.44852787256240845 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5512820482254028 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5163424015045166 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5583860278129578 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2748015820980072 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5545635223388672 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 *”²ù Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.552556037902832 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_5‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3185184895992279 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23563113808631897 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.314353346824646 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04158986359834671 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8516348004341125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9986394643783569 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.44963526725769043 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5102732181549072 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6754385828971863 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5359246134757996 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013197427615523338 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5447530746459961 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137115478515625 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2870831787586212 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5315065979957581 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5621899962425232 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5455687642097473 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.037176571786403656 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4127906858921051 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1807130128145218 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5445402264595032 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3697747588157654 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2119491994380951 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18345864117145538 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018260840326547623 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14367815852165222 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5604895949363708 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1860005259513855 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878787517547607 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_7‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 particle/accuracy_1_of_5‚1 +particle/specificity‚0.9510868191719055 jet/accuracy_1_of_6‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9895887970924377 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.44685104489326477 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3611111044883728 particle/accuracy_2_of_6‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5034080743789673 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27288973331451416 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5185185074806213 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1595744639635086 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27776098251342773 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5430377721786499 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_6‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27097123861312866 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_7‚1 particle/accuracy_7_of_7‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895178079605103 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.42711541056632996 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_3_of_4‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18399354815483093 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9926888942718506 +particle/sensitivity‚0.3141602575778961 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5253510475158691 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 jet/accuracy_1_of_7‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08199996501207352 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40334826707839966 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6062595844268799 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20747126638889313 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 'particle/f_score‚0.3978305459022522 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22453705966472626 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.53570955991745 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4315440356731415 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.20631510019302368 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9871737957000732 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_6‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09056513011455536 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5178834795951843 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6204860806465149 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.33712121844291687 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.43703702092170715 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.03846153989434242 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16854242980480194 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 epoch‚8 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9909262657165527 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28169742226600647 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166567325592 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1953963190317154 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 particle/accuracy_1_of_3‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 % _timestamp‚1.7884365785068126e+09 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2589128315448761 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3965201675891876 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5813784599304199 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4435223937034607 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.008064515888690948 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5500505566596985 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 5 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.52923184633255 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17142947018146515 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5857216119766235 ( Purity/1FRt3SRqqt1FBt/FRt_purity‚0.5 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 particle/accuracy_5_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27955180406570435 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4583333432674408 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5453793406486511 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17642934620380402 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2153647243976593 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39791667461395264 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.005397115834057331 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.49928638339042664 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5970278978347778 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1640309989452362 +jet/accuracy_2_of_3‚0.26016706228256226 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.44869279861450195 particle/accuracy_1_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.540717363357544 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8178004026412964 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5731286406517029 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333432674408 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9865562319755554 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26519501209259033 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01744186133146286 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5154428482055664 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.23425929248332977 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03099624067544937 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9783282279968262 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4340278208255768 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5230268836021423 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2361111044883728 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5646774768829346 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4915153682231903 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006163251120597124 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚41.52708907171866 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7071903347969055 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4289272129535675 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5075371265411377 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5627439022064209 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 +jet/accuracy_2_of_2‚0.11454810947179794 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.529420018196106 particle/accuracy_4_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9901536107063293 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08482784032821655 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5422271490097046 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 trainer/global_step‚10880 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23648647964000702 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 *jet/accuracy_1_of_3‚0.8782657980918884 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5902392864227295 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 +jet/accuracy_3_of_4‚0.07361501455307007 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26436781883239746 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07758620381355286 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_2_of_7‚0.5 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333432674408 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18394441902637482 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10640041530132294 jet/accuracy_3_of_7‚0 particle/accuracy_3_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10576117038726807 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9913495779037476 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10809538513422012 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_1_of_2‚0.773730993270874 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121212482452393 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4614379107952118 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998635470867157 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34702378511428833 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3857503831386566 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1247267797589302 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18787887692451477 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.029428070411086082 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_1_of_1‚0.499502956867218 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.992340087890625 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23950017988681793 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.190476194024086 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20940664410591125 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6111111044883728 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5308223366737366 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02099432982504368 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0478997640311718 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4197531044483185 *jet/accuracy_3_of_6‚0.1538461595773697 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36069172620773315 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.37057727575302124 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18738901615142822 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983899593353271 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9766522645950317 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1688069850206375 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2984972298145294 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1739766001701355 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15111111104488373 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_6‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21483851969242096 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.641080379486084 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.181843563914299 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878621697425842 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.529420018196106 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.181843563914299 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5881928205490112 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.019736841320991516 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_2_of_3‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16540473699569702 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3236176669597626 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4319688081741333 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9272766709327698 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9923369288444519 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5340144038200378 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.203125 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 jet/accuracy_7_of_7‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19997796416282654 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9848790168762207 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 _step‚894 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.44780099391937256 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6075549721717834 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5199999809265137 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.43051478266716003 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1800329089164734 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.561073362827301 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_4_of_6‚1 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07372057437896729 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.007547169923782349 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5023846626281738 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20557819306850433 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4780701994895935 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 _runtime‚8263.213979561 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03067384660243988 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9858967065811157 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42013075947761536 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0052910055965185165 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5515188574790955 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.026171045377850533 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15789473056793213 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48793351650238037 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9686188101768494 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 à•=ú 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5275380611419678 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5880952477455139 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5022637248039246 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4958333671092987 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 particle/accuracy_1_of_4‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17038963735103607 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23141691088676453 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1358024626970291 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011944783851504326 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09196428209543228 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.431081086397171 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13619349896907806 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_6‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3742063641548157 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14861109852790833 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15494228899478912 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17696425318717957 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.365727037191391 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4700262248516083 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0þ.zÄÍ‘¾'‚(:† äÀåÔý¼¯ gpu.0.gpu‚5.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚30.0 gpu.0.powerWatts‚76.196' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚15.2392 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p¢T÷.¿'‚(:£ äÀåÔ†íû disk.sda3.out‚ 332.3515625 network.sent‚ 2.6276875e+07) disk./.usagePercent‚73.92521227325514 disk.sda.in‚0.5 disk.sda1.out‚0$ disk.nvme0n1.out‚49091.958984375# disk.nvme0c0n1.in‚ 1071.99609375 disk.dm-1.out‚ 332.3515625& disk.nvme0c0n1.out‚49091.958984375 disk.rbd1.out‚ 4.29296875% disk./.usageGB‚1101.5390853881836 disk.rbd1.in‚0 network.recv‚ 2.815728e+06 disk.dm-1.in‚0.5 disk.sda2.out‚0* proc.memory.percent‚20.418167114257812 disk.sda1.in‚0! disk.dm-0.out‚49091.958984375$ memory_percent‚99.96182918548584& proc.memory.availableMB‚ 6.25390625 disk.rbd2.in‚0! disk.rbd2.out‚219.10107421875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.in‚0.5 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1071.99609375 proc.memory.rssMB‚ 3345.3125 cpu‚25.370842317182234 disk.sda.out‚ 332.3515625! disk.nvme0n1.in‚ 1071.99609375 disk.sda2.in‚0 disk.rbd0.in‚0ß…žÀ'‚(:“ óÀåÔ¬øØ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚141.078' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.215600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶·é5Á'‚(:ª óÀåÔ—ÿ£‘ disk.sda.out‚ 333.140625 disk.rbd1.out‚ 4.29296875! disk.rbd2.out‚219.24169921875 disk.drbd1777.out‚0 network.recv‚ 2.822565e+06& disk.nvme0n1.out‚49565.75439453125 disk.rbd0.out‚0 disk.rbd2.in‚0# disk.dm-0.out‚49565.75439453125# disk.nvme0c0n1.in‚ 1071.99609375 disk.sda2.in‚0 cpu‚25.370020218412723% disk./.usageGB‚1100.8262214660645( disk.nvme0c0n1.out‚49565.75439453125 disk.rbd3.out‚0# memory_percent‚91.2874698638916 network.sent‚ 2.6411459e+07 disk.dm-0.in‚ 1071.99609375 disk.dm-1.in‚0.5! disk.nvme0n1.in‚ 1071.99609375 disk.rbd0.in‚0 disk.drbd1777.in‚0" proc.memory.rssMB‚ 2872.3515625 disk.sda.in‚0.5 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1427.4609375* proc.memory.percent‚17.531442642211914 proc.cpu.threads‚17) disk./.usagePercent‚73.87737137730906 disk.sda1.in‚0 disk.dm-1.out‚ 333.140625 disk.sda3.in‚0.5 disk.sda3.out‚ 333.140625 disk.rbd3.in‚0 ±Ræ2Â'- _wandbruntime‚8280  _runtime‚8280ÓíûÆÃ'À % lr-AdamW/pg1‚6.63354360236404e-06 % lr-AdamW/pg2‚6.63354360236404e-06 trainer/global_step‚10899 % _timestamp‚1.7884365956531222e+09 _runtime‚8280.357889013 _step‚895â¥ßVËÄ'Å % lr-AdamW/pg1‚6.63354360236404e-06 % lr-AdamW/pg2‚6.63354360236404e-06 trainer/global_step‚10899 % _timestamp‚1.7884365956531222e+09 _runtime‚8280.357889013 _step‚895ÿöp8ü2Å'- _wandbruntime‚8280  _runtime‚8280@m¼è'Æ'âO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52271382 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚693489236745728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9141834484224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9141834484224 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144995629 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9108172400030 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9142490846720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9589834004480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚644289478739968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚703079070750208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚627591111999258 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144996052 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚635147662258688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92724670 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364940415 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206051455 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206051455 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚8280.358950463 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158908998 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚421 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚703078881089024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92724668 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52270961 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚635147644255744 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚618482268122112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚644289478739968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 364960455 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693489212805632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚618482285161472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚635147644255744 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚618482268122112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚644289478739968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 364960455 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693489212805632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚618482285161472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9142490846720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9589834004480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚644289478739968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚703079070750208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚627591111999258 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144996052 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚635147662258688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92724670 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364940415 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206051455 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206051455 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚8280.358950463 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158908998 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚421 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚703078881089024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414€¤2È'- _wandbruntime‚8281  _runtime‚8281]Y¸É'ìM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158923016 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚627646161087666 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206069935 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚8281.020222381 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92732838 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚703141387032064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9108988834994 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9143310287872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9142653912064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9143310287872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364972885 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145008264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚635204015297536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 364992951 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚635204015297536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884365963148496e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92732836 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚618537189292032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚703141580874240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364992951 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9109643272958 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚644347325585408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 QCDeviceStatsMo×Ö‡¸Õnitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9590693390848 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚644346651206656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52275428 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158923014 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚635203997294592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9590523488768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52275850 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚422 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚693550887483392 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚635203997294592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145008688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206049871 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 trainer/global_step‚10899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚618537172252672 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145008264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚635204015297536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 364992951 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚635204015297536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884365963148496e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92732836 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚618537189292032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚703141580874240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364992951 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9109643272958 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚644347325585408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9590693390848 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚644346651206656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52275428 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158923014 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚635203997294592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9590523488768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52275850 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚422 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚693550887483392 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚635203997294592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145008688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206049871 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 trainer/global_step‚10899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚618537172252672 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚627646161087666 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206069935 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚8281.020222381 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92732838 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚703141387032064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9108988834994 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9143310287872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9142653912064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9143310287872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364972885 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521ÐøÚ2Ë'- _wandbruntime‚8281  _runtime‚8281ƒâ}CZÌ'Ô 0loss/SRqqt1/assignment_loss‚0.90739154206988 2loss/SRqqt2/assignment_loss‚1.2088703848421574 2loss/SRqqt3/assignment_loss‚1.6929116401916895 /loss/FRt1/detection_loss‚1.5833681822433823 /loss/FRt4/detection_loss‚2.2230773509844526 /loss/FBt1/detection_loss‚0.5145125048398808 _runtime‚8281.021323125 1loss/FBt3/assignment_loss‚0.46644827550543205 .loss/FBt2/detection_loss‚2.250750557942824 .loss/FBt3/detection_loss‚2.742735343320029 1loss/FBt1/assignment_loss‚0.08643485730574821 1loss/FBt2/assignment_loss‚0.22454917585491785 epoch‚9 /loss/FRt4/assignment_loss‚2.873921794735867 1loss/FBt4/assignment_loss‚0.07880504784407094 $ _timestamp‚1.788436596316662e+09 2loss/SRqqt4/assignment_loss‚1.4017449158224433 1loss/SRqqt2/detection_loss‚1.7161503336867507 .loss/FRt2/detection_loss‚2.359208085064618 1loss/SRqqt1/detection_loss‚1.0388659297887768 0loss/SRqqt3/detection_loss‚2.211192092834375 1loss/SRqqt4/detection_loss‚2.0543896897385516 .loss/FBt4/detection_loss‚2.259504795074463 0loss/FRt1/assignment_loss‚1.3344071937389061 0loss/FRt2/assignment_loss‚2.4701010854739063 /loss/FRt3/assignment_loss‚3.533922689301627 /loss/FRt3/detection_loss‚2.7442675488335744 %loss/total_loss‚39.97753101703833 _step‚898 trainer/global_step‚10899œzVÙ_Í'Ù .loss/FBt4/detection_loss‚2.259504795074463 0loss/FRt1/assignment_loss‚1.3344071937389061 0loss/FRt2/assignment_loss‚2.4701010854739063 /loss/FRt3/assignment_loss‚3.533922689301627 /loss/FRt3/detection_loss‚2.7442675488335744 %loss/total_loss‚39.97753101703833 _step‚898 trainer/global_step‚10899 0loss/SRqqt1/assignment_loss‚0.90739154206988 2loss/SRqqt2/assignment_loss‚1.2088703848421574 2loss/SRqqt3/assignment_loss‚1.6929116401916895 /loss/FRt1/detection_loss‚1.5833681822433823 /loss/FRt4/detection_loss‚2.2230773509844526 /loss/FBt1/detection_loss‚0.5145125048398808 _runtime‚8281.021323125 1loss/FBt3/assignment_loss‚0.46644827550543205 .loss/FBt2/detection_loss‚2.250750557942824 .loss/FBt3/detection_loss‚2.742735343320029 1loss/FBt1/assignment_loss‚0.08643485730574821 1loss/FBt2/assignment_loss‚0.22454917585491785 epoch‚9 /loss/FRt4/assignment_loss‚2.873921794735867 1loss/FBt4/assignment_loss‚0.07880504784407094 $ _timestamp‚1.788436596316662e+09 2loss/SRqqt4/assignment_loss‚1.4017449158224433 1loss/SRqqt2/detection_loss‚1.7161503336867507 .loss/FRt2/detection_loss‚2.359208085064618 1loss/SRqqt1/detection_loss‚1.0388659297887768 0loss/SRqqt3/detection_loss‚2.211192092834375 1loss/SRqqt4/detection_loss‚2.0543896897385516‚†AÝEÎ'‚(:’ ‚ÁåÔ´ñÞ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.89' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.177999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o;Ç2Ï'‚(:§ ‚ÁåԜд# disk.dm-0.out‚50138.08740234375 disk.sda.in‚0.5 disk.dm-1.out‚ 333.65625% disk./.usageGB‚1100.8243408203125 disk.drbd1777.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1427.4296875 network.recv‚ 2.826605e+06 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 4.29296875 proc.cpu.threads‚17) disk./.usagePercent‚73.87724516559469! disk.nvme0n1.in‚ 1071.99609375 disk.rbd1.in‚0( disk.nvme0c0n1.out‚50138.08740234375 disk.sda2.out‚0$ memory_percent‚91.28766059875488 network.sent‚ 2.6416626e+07 disk.rbd2.in‚0! disk.rbd2.out‚219.25341796875 disk.rbd3.out‚0 disk.dm-0.in‚ 1071.99609375 disk.sda1.in‚0 disk.dm-1.in‚0.5" proc.memory.rssMB‚ 2872.3515625 disk.rbd0.in‚0 disk.sda.out‚ 333.65625& disk.nvme0n1.out‚50138.08740234375# disk.nvme0c0n1.in‚ 1071.99609375* proc.memory.percent‚17.531442642211914 disk.rbd0.out‚0 disk.sda3.in‚0.5 disk.sda3.out‚ 333.65625 disk.sda2.in‚0 cpu‚25.36941875135872Í^Ó“Ð'‚(:ˆ ‘ÁåÔ²÷Ù gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚141.211' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.2422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡!K4Ñ'‚(:© ‘ÁåÔÛÈÍ‘ disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda.in‚0.5 network.recv‚ 2.83324e+06 disk.drbd1777.in‚0 disk.sda1.out‚0# disk.dm-0.out‚50239.56689453125 network.sent‚ 2.6525389e+07% disk./.usageGB‚1100.8242797851562 disk.sda2.in‚0 disk.rbd2.out‚ 219.40625 disk.rbd3.out‚0 disk.dm-0.in‚ 1071.99609375$ memory_percent‚91.28766059875488 disk.sda.out‚ 334.47265625) disk./.usagePercent‚73.87724106947414 disk.dm-1.out‚ 334.47265625! disk.nvme0n1.in‚ 1071.99609375 disk.drbd1777.out‚0* proc.memory.percent‚17.531442642211914# disk.nvme0c0n1.in‚ 1071.99609375 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.5 disk.sda3.out‚ 334.47265625" proc.memory.rssMB‚ 2872.3515625 proc.cpu.threads‚17 disk.rbd1.out‚ 4.29296875( disk.nvme0c0n1.out‚50239.56689453125 disk.sda3.in‚0.5 disk.rbd3.in‚0( proc.memory.availableMB‚ 1427.4296875 cpu‚25.36886647866944& disk.nvme0n1.out‚50239.56689453125 disk.rbd2.in‚0 disk.sda1.in‚0Æj­n2Ò'- _wandbruntime‚8316  _runtime‚8316)‹JÆÓ'À _runtime‚ 8316.1098981 _step‚899 & lr-AdamW/pg1‚6.093205066817566e-06 & lr-AdamW/pg2‚6.093205066817566e-06 trainer/global_step‚10949 % _timestamp‚1.7884366314051113e+09¨£ eËÔ'Å trainer/global_step‚10949 % _timestamp‚1.7884366314051113e+09 _runtime‚ 8316.1098981 _step‚899 & lr-AdamW/pg1‚6.093205066817566e-06 & lr-AdamW/pg2‚6.093205066817566e-06ƒ=«Çt2Õ'- _wandbruntime‚8316  _runtime‚8316!çxšè'Ö'âO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159609798 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145628936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚696571773628928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚630376482389550 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9149007739438 trainer/global_step‚10949 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366587307 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚637965296198144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚696571749688832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647148790853632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚621227491689472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206957469 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93133068 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637965314201088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚706204560523264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9632786894336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145628495 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚637965296198144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚900 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚647148116488192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚637965314201088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647148116488192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9182820290048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9182820290048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚630377153866666 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9149662177194 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 % _timestamp‚1.7884366314059412e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159609798 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚621227474650112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366567267 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9183476652544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52495866 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206977507 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9183476652544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872â‘[m×'çO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647148790853632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚621227491689472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =7DeviceStatsMonitorºpkÖ€ .on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206957469 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93133068 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637965314201088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚706204560523264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9632786894336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145628495 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚637965296198144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚900 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚647148116488192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚637965314201088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647148116488192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9182820290048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9182820290048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚630377153866666 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9149662177194 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 % _timestamp‚1.7884366314059412e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159609798 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚621227474650112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366567267 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9183476652544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52495866 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206977507 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9183476652544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159609798 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145628936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚696571773628928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚630376482389550 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9149007739438 trainer/global_step‚10949 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366587307 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚637965296198144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚696571749688832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944„¿_‹ 2Ø'- _wandbruntime‚8316  _runtime‚8316êžØ)ò&Ù'ìM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 366599737 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚10949 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 366619803 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚638021667239936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647205963333632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52500348 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚8316.775348401 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366619803 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159623816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206975923 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145641141 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9633454835712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93141238 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚621282395820032 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159623814 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206995987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647205288954880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚630432874432398 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9149824174402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9183639717888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9633631029248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚901 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145641586 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0JÒÃe÷&Ú'ñM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159623816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206975923 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145641141 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9633454835712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93141238 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚621282395820032 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159623814 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206995987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647205288954880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚630432874432398 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9149824174402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9183639717888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9633631029248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚901 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145641586 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 366599737 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚10949 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 366619803 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚638021667239936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647205963333632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52500348 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚8316.775348401 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366619803 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844…Ë(@2Û'- _wandbruntime‚8316  _runtime‚8316y«ïÈVÜ'Ð 1loss/SRqqt2/assignment_loss‚1.378272358411923 0loss/FBt2/assignment_loss‚0.5537798083769303 0loss/FBt4/assignment_loss‚0.3950549103319645 /loss/FRt2/detection_loss‚2.3916381411254406 1loss/SRqqt4/detection_loss‚1.5108363721519709 /loss/FRt2/assignment_loss‚2.354274110496044 0loss/FBt1/assignment_loss‚0.1927673965409043 /loss/FBt1/detection_loss‚0.6774708456359804 1loss/SRqqt2/detection_loss‚1.6759692645072937 /loss/FBt3/detection_loss‚2.5733222828971014 %loss/total_loss‚41.51295562271541 _runtime‚8316.776510952 /loss/FRt3/assignment_loss‚2.963050375724661 .loss/FRt1/detection_loss‚1.558198768645525 /loss/FRt3/detection_loss‚2.7039386260098426 0loss/SRqqt1/detection_loss‚0.785799710595837 /loss/SRqqt3/detection_loss‚2.68051538509982 epoch‚9 trainer/global_step‚10949 0loss/FRt1/assignment_loss‚1.3577396490357139 2loss/SRqqt1/assignment_loss‚0.9395940427381831 1loss/SRqqt3/assignment_loss‚1.847131249960512 0loss/FBt3/assignment_loss‚0.3743787971058434 .loss/FRt4/detection_loss‚2.412015026807785 % _timestamp‚1.7884366320718231e+09 .loss/FBt4/detection_loss‚3.572978210449219 2loss/SRqqt4/assignment_loss‚1.1170017764088698 /loss/FBt2/detection_loss‚1.8758776560425758 _step‚902 0loss/FRt4/assignment_loss‚3.6213508576154707—z}˜[Ý'Õ /loss/FBt2/detection_loss‚1.8758776560425758 _step‚902 0loss/FRt4/assignment_loss‚3.6213508576154707 1loss/SRqqt2/assignment_loss‚1.378272358411923 0loss/FBt2/assignment_loss‚0.5537798083769303 0loss/FBt4/assignment_loss‚0.3950549103319645 /loss/FRt2/detection_loss‚2.3916381411254406 1loss/SRqqt4/detection_loss‚1.5108363721519709 /loss/FRt2/assignment_loss‚2.354274110496044 0loss/FBt1/assignment_loss‚0.1927673965409043 /loss/FBt1/detection_loss‚0.6774708456359804 1loss/SRqqt2/detection_loss‚1.6759692645072937 /loss/FBt3/detection_loss‚2.5733222828971014 %loss/total_loss‚41.51295562271541 _runtime‚8316.776510952 /loss/FRt3/assignment_loss‚2.963050375724661 .loss/FRt1/detection_loss‚1.558198768645525 /loss/FRt3/detection_loss‚2.7039386260098426 0loss/SRqqt1/detection_loss‚0.785799710595837 /loss/SRqqt3/detection_loss‚2.68051538509982 epoch‚9 trainer/global_step‚10949 0loss/FRt1/assignment_loss‚1.3577396490357139 2loss/SRqqt1/assignment_loss‚0.9395940427381831 1loss/SRqqt3/assignment_loss‚1.847131249960512 0loss/FBt3/assignment_loss‚0.3743787971058434 .loss/FRt4/detection_loss‚2.412015026807785 % _timestamp‚1.7884366320718231e+09 .loss/FBt4/detection_loss‚3.572978210449219 2loss/SRqqt4/assignment_loss‚1.1170017764088698†üY¼H’Þ'‚(:‡  ÁåÔÈ®•Ž gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚98.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.6232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒá–ù2ß'‚(:§  ÁåÔÏü’& disk.nvme0n1.out‚50339.50927734375 disk.sda2.in‚0 disk.rbd2.out‚ 219.40625 disk.drbd1777.in‚0 disk.dm-1.out‚ 335.5078125( proc.memory.availableMB‚ 1427.3828125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1073.82421875 disk.rbd2.in‚0 disk.sda3.out‚ 335.5078125% disk./.usageGB‚1100.8226013183594! disk.nvme0n1.in‚ 1073.82421875 disk.sda2.out‚0 disk.dm-0.in‚ 1073.82421875 cpu‚25.368290879651802( disk.nvme0c0n1.out‚50339.50927734375 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 2.837224e+06 disk.rbd0.in‚0 disk.sda.in‚0.5 disk.sda.out‚ 335.5078125 disk.dm-1.in‚0.5" proc.memory.rssMB‚ 2872.3515625 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.29296875 disk.drbd1777.out‚0 disk.rbd3.in‚0# memory_percent‚91.2879467010498 disk.sda3.in‚0.5 disk.rbd3.out‚0# disk.dm-0.out‚50339.50927734375) disk./.usagePercent‚73.87712842615906* proc.memory.percent‚17.531442642211914 network.sent‚ 2.6530844e+07–*·xà'‚(:“ ¯ÁåÔÎòÚ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0Så&.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚161.427' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.285399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â&I4á'‚(:© ¯ÁåÔ™š‘ disk.sda.out‚ 336.07421875 disk.rbd1.in‚0 disk.rbd1.out‚ 4.29296875" disk.nvme0c0n1.in‚ 1074.5859375 disk.sda1.in‚0 disk.dm-1.in‚0.5 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2872.3515625* proc.memory.percent‚17.531442642211914 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.dm-1.out‚ 336.07421875 disk.nvme0n1.in‚ 1074.5859375% disk.nvme0n1.out‚50443.2177734375% disk./.usageGB‚1100.8236198425293 disk.rbd2.in‚0) proc.memory.availableMB‚ 1427.39453125 network.recv‚ 2.843996e+06 disk.rbd3.out‚0 disk.sda.in‚0.5 disk.sda1.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚73.87719678017072" disk.dm-0.out‚50443.2177734375' disk.nvme0c0n1.out‚50443.2177734375 disk.rbd0.in‚0 disk.sda3.in‚0.5 network.sent‚ 2.6639785e+07 disk.rbd2.out‚ 221.12109375 cpu‚25.367725555915676 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 1074.5859375 disk.sda3.out‚ 336.07421875$ memory_percent‚91.28787517547607ú©2â'- _wandbruntime‚8351  _runtime‚8351eBƒ›Æã'À trainer/global_step‚10999 % _timestamp‚1.7884366672558231e+09 _runtime‚8351.960540744 _step‚903 % lr-AdamW/pg1‚5.57521064309244e-06 % lr-AdamW/pg2‚5.57521064309244e-06þ[LKËä'Å % lr-AdamW/pg1‚5.57521064309244e-06 % lr-AdamW/pg2‚5.57521064309244e-06 trainer/global_step‚10999 % _timestamp‚1.7884366672558231e+09 _runtime‚8351.960540744 _step‚903‡t;Ã2å'- _wandbruntime‚8351  _runtime‚8351ß.Vêç'æ'áO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚640782966143488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9189843078846 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207903559 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9675589388800 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚633162524256958 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚650007428601856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚10999 _runtime‚8351.961491926 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9224462458368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9223806095872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52719266 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9223806095872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 207903559 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52718844 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚709329899900928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9190497516602 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚650006754236416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 368194119 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650006754236416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚699654286572032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9675423667712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146260736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93541470 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160310598 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207883521 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚623972681178112<<ì'ç'æO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 160310600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚699654310512128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚640782948140544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 160310600 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚640782966143488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚640782948140544 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚650007428601856 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146260312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 $ _timestamp‚1.788436667256543e+09 _step‚904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚640782966143488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9189843078846 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207903559 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9675589388800 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚633162524256958 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚650007428601856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚10999 _runtime‚8351.961491926 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9224462458368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9223806095872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52719266 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9223806095872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 207903559 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52718844 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚709329899900928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9190497516602 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚650006754236416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 368194119 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650006754236416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚699654286572032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9675423667712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146260736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93541470 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160310598 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207883521 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚623972681178112 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9225281899520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146272919 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52723724 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207901975 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚650064601081856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207922039 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160324614 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160324614 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9676277213184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884366679269502e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93549636 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160324616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93549638 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368246655 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9676447115264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚709392214522880 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 368226589 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚650064601081856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚650063926703104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚640839301179392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9224625523712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 368246655 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0t2÷¦mê'ñM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160324616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool`ÇŠ%.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93549638 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368246655 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9676447115264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚709392214522880 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 368226589 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚650064601081856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚650063926703104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚640839301179392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9224625523712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 368246655 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚709392408365056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚640839319182336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚624027585308672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9190659513810 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚8352.632207088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9225281899520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146272919 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52723724 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207901975 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚650064601081856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207922039 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160324614 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160324614 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9676277213184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884366679269502e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93549636‰ò¤¶ 2ë'- _wandbruntime‚8352  _runtime‚8352¢Ä¡?[ì'Õ 2loss/SRqqt3/assignment_loss‚1.7397035729140042 1loss/FBt1/assignment_loss‚0.12185386516759589 /loss/FBt2/detection_loss‚2.4590671116655525 _runtime‚8352.633438378 1loss/SRqqt1/assignment_loss‚0.700425786510683 1loss/FBt2/assignment_loss‚0.27092405524902835 .loss/FRt1/detection_loss‚1.793197829897205 1loss/SRqqt3/detection_loss‚2.0202998287975786 trainer/global_step‚10999 _step‚906 0loss/FRt1/assignment_loss‚1.1703982551892598 1loss/FBt4/assignment_loss‚0.24269622473511845 /loss/FBt3/detection_loss‚3.3180052723203386 epoch‚9 .loss/FRt3/detection_loss‚2.852598015604348 1loss/SRqqt2/detection_loss‚1.7767025296594583 0loss/FBt1/detection_loss‚0.40863805488224464 0loss/FRt3/assignment_loss‚3.0702358730908097 /loss/FRt2/detection_loss‚2.3469426564872267 -loss/FRt4/detection_loss‚2.18827711695282 .loss/FBt4/detection_loss‚2.176319569349289 1loss/SRqqt2/assignment_loss‚1.381125604916437 2loss/SRqqt4/assignment_loss‚1.2711789109843208 0loss/SRqqt1/detection_loss‚1.102067939707866 1loss/SRqqt4/detection_loss‚1.8079216350266274 /loss/FRt4/assignment_loss‚2.915538401175768 0loss/FBt3/assignment_loss‚0.0961801616649609 % _timestamp‚1.7884366679287639e+09 %loss/total_loss‚39.47893151211187 0loss/FRt2/assignment_loss‚2.2486332401633264ʉf\`í'Ú %loss/total_loss‚39.47893151211187 0loss/FRt2/assignment_loss‚2.2486332401633264 2loss/SRqqt3/assignment_loss‚1.7397035729140042 1loss/FBt1/assignment_loss‚0.12185386516759589 /loss/FBt2/detection_loss‚2.4590671116655525 _runtime‚8352.633438378 1loss/SRqqt1/assignment_loss‚0.700425786510683 1loss/FBt2/assignment_loss‚0.27092405524902835 .loss/FRt1/detection_loss‚1.793197829897205 1loss/SRqqt3/detection_loss‚2.0202998287975786 trainer/global_step‚10999 _step‚906 0loss/FRt1/assignment_loss‚1.1703982551892598 1loss/FBt4/assignment_loss‚0.24269622473511845 /loss/FBt3/detection_loss‚3.3180052723203386 epoch‚9 .loss/FRt3/detection_loss‚2.852598015604348 1loss/SRqqt2/detection_loss‚1.7767025296594583 0loss/FBt1/detection_loss‚0.40863805488224464 0loss/FRt3/assignment_loss‚3.0702358730908097 /loss/FRt2/detection_loss‚2.3469426564872267 -loss/FRt4/detection_loss‚2.18827711695282 .loss/FBt4/detection_loss‚2.176319569349289 1loss/SRqqt2/assignment_loss‚1.381125604916437 2loss/SRqqt4/assignment_loss‚1.2711789109843208 0loss/SRqqt1/detection_loss‚1.102067939707866 1loss/SRqqt4/detection_loss‚1.8079216350266274 /loss/FRt4/assignment_loss‚2.915538401175768 0loss/FBt3/assignment_loss‚0.0961801616649609 % _timestamp‚1.7884366679287639e+09Š I;{’î'‚(:‡ ¾ÁåÔŠº¡ gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚90.786' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…4ï'‚(:© ¾ÁåÔžˆ­ network.recv‚ 2.847887e+06 disk.rbd1.out‚ 4.29296875 disk.dm-0.in‚ 1075.390625* proc.memory.percent‚17.531442642211914 disk.nvme0n1.in‚ 1075.390625) disk./.usagePercent‚73.87725540589607 disk.dm-1.in‚0.5 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.5 disk.rbd3.out‚0# disk.dm-0.out‚50444.53271484375$ memory_percent‚91.29135608673096 disk.rbd1.in‚0 disk.sda.in‚0.5 disk.sda.out‚ 336.69140625! disk.nvme0c0n1.in‚ 1075.390625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 221.12109375 network.sent‚ 2.6645059e+07& disk.nvme0n1.out‚50444.53271484375 disk.sda3.out‚ 336.69140625) proc.memory.availableMB‚ 1426.82421875 cpu‚25.367162204456836 proc.cpu.threads‚17 disk.rbd3.in‚0% disk./.usageGB‚1100.8244934082031 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 336.69140625( disk.nvme0c0n1.out‚50444.53271484375 disk.rbd0.in‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2872.3515625²çx0’ð'‚(:‡ ÍÁåÔ¯õù gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚162.065' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.413 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤O3ñ'‚(:¨ ÍÁåÔé³ã‘ disk.drbd1777.in‚0 disk.dm-0.in‚ 1075.73046875 disk.sda1.out‚0 disk.dm-1.out‚ 337.546875* proc.memory.percent‚17.531442642211914 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0.5! disk.nvme0n1.in‚ 1075.73046875( disk./.usagePercent‚73.8772595020166 disk.sda3.out‚ 337.546875& disk.nvme0n1.out‚50544.82958984375" proc.memory.rssMB‚ 2872.3515625% disk./.usageGB‚1100.8245544433594 disk.sda2.in‚0# disk.dm-0.out‚50544.82958984375 cpu‚25.366616365521914 proc.cpu.threads‚17 disk.sda3.in‚0.5 disk.drbd1777.out‚0 disk.sda.in‚0.5 disk.rbd2.out‚ 221.22265625 network.recv‚ 2.854345e+06# disk.nvme0c0n1.in‚ 1075.73046875) proc.memory.availableMB‚ 1426.68359375( disk.nvme0c0n1.out‚50544.82958984375 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 2.6753755e+07 disk.rbd1.in‚0 disk.rbd1.out‚ 4.29296875 disk.rbd0.in‚0 disk.sda.out‚ 337.546875$ memory_percent‚91.29221439361572 disk.rbd0.out‚06ïfò'‚(:’ ÜÁåÔ¤ùˆŽ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚160.07' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.013999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬‰fëó'‚(: ÜÁåԜƿ disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.out‚338( proc.memory.availableMB‚ 1426.1484375( disk.nvme0c0n1.out‚50644.19677734375( disk./.usagePercent‚73.8773342562166 disk.dm-1.in‚0.5! disk.nvme0n1.in‚ 1076.44921875 disk.rbd1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.531442642211914 cpu‚25.36605976941282& disk.nvme0n1.out‚50644.19677734375 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚338 disk.sda.in‚0.5 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 221.22265625 disk.drbd1777.in‚0# disk.dm-0.out‚50644.19677734375 network.recv‚ 2.858313e+06 disk.sda1.in‚0 network.sent‚ 2.6759377e+07# disk.nvme0c0n1.in‚ 1076.44921875 disk.sda3.in‚0.5" proc.memory.rssMB‚ 2872.3515625$ disk./.usageGB‚1100.825668334961$ memory_percent‚91.29548072814941 disk.dm-0.in‚ 1076.44921875 disk.dm-1.out‚338 disk.rbd1.out‚ 4.29296875Tm;„2ô'- _wandbruntime‚8388  _runtime‚8388¿RŠ‘Èõ' _step‚907 & lr-AdamW/pg1‚5.079668280450172e-06 & lr-AdamW/pg2‚5.079668280450172e-06 trainer/global_step‚11049 % _timestamp‚1.7884367033448415e+09 _runtime‚8388.049609534 ÃþÝÍö'Ç & lr-AdamW/pg1‚5.079668280450172e-06 & lr-AdamW/pg2‚5.079668280450172e-06 trainer/global_step‚11049 % _timestamp‚1.7884367033448415e+09 _runtime‚8388.049609534 _step‚907‹ ¢Î2÷'- _wandbruntime‚8388  _runtime‚8388¶Ùè'ø'âO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146892936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369820971 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93949870 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚643600600082944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚652866066350080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635948566124366 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161011398 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚652865391984640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52942628 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9230678418254 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208829611 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚643600618085888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369841011 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161011400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884367033456316e+09 _step‚908 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161011400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚712455301759488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208809573 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93949868 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9264791901696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9265448264192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52943066 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702736847395328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9231332856010 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9264791901696£{ùùõù'çO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9230678418254 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208829611 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚643600618085888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369841011 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161011400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884367033456316e+09 _step‚908 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161011400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚712455301759488 ICDeviceStatsMonitor.on_train_batJè%øch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208809573 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93949868 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9264791901696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9265448264192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52943066 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702736847395328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9231332856010 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9264791901696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146892936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369820971 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93949870 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚643600600082944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚652866066350080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635948566124366 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161011398 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚652865391984640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52942628 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025Œ¢îî°2ú'- _wandbruntime‚8388  _runtime‚8388j‹€ñ&û'ëM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚643656953121792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369873507 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚652922564451328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 369853441 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚712517629952512 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161025414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚643656971124736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚8388.733557587 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146905058 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚626772791836672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208848091 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369853441 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚425 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702798474192896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9266267705344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208848091 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52947022 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9232149291182 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚652923238830080 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208828027 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9265611329536 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9265611329536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚712517813308928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚626772808876032 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚636004286689890 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9266267705344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9719315175936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788436704028247e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702798498132992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944à¹F9ö&ü'ðM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚652922564451328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 369853441 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚712517629952512 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161025414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚643656971124736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚8388.733557587 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146905058 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚626772791836672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208848091 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369853441 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚425 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702798474192896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9266267705344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208848091 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52947022 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9232149291182 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚652923238830080 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208828027 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9265611329536 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9265611329536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚712517813308928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚626772808876032 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚636004286689890 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9266267705344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9719315175936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788436704028247e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702798498132992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚643656953121792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369873507 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0Y/„ª2ý'- _wandbruntime‚8388  _runtime‚8388lÈŠë[þ'Õ 2loss/SRqqt3/assignment_loss‚1.6348996154847555 1loss/FBt1/assignment_loss‚0.16558069305189724 1loss/SRqqt4/detection_loss‚1.5878410466015338 /loss/FBt2/detection_loss‚2.0240909915703993 %loss/total_loss‚40.24055009741805 trainer/global_step‚11049 /loss/FRt2/assignment_loss‚2.508428845140669 1loss/SRqqt4/assignment_loss‚1.313946454351147 0loss/FBt3/assignment_loss‚0.6560337394825183 epoch‚9 .loss/FRt4/detection_loss‚2.286000685982926 1loss/SRqqt3/detection_loss‚2.4429666493088007 % _timestamp‚1.7884367040300946e+09 _step‚910 /loss/FRt4/assignment_loss‚2.782587183076282 /loss/FRt1/detection_loss‚1.6498327692772479 /loss/FRt3/detection_loss‚3.1337150988670497 0loss/SRqqt1/detection_loss‚0.895592894577063 .loss/FBt4/detection_loss‚3.107461074988047 0loss/FRt1/assignment_loss‚1.1964442818722827 0loss/FRt3/assignment_loss‚3.0283138064237742 1loss/SRqqt2/assignment_loss‚1.273472949298468 1loss/FBt2/assignment_loss‚0.33167818256102777 /loss/FBt1/detection_loss‚0.5234598454747863 _runtime‚8388.734745485 2loss/SRqqt1/assignment_loss‚0.8299342901660846 0loss/FBt4/assignment_loss‚0.3834845603754123 /loss/FRt2/detection_loss‚2.5432119551632137 1loss/SRqqt2/detection_loss‚1.4194160907946785 /loss/FBt3/detection_loss‚2.5221563935279847¾v%G`ÿ'Ú /loss/FRt4/assignment_loss‚2.782587183076282 /loss/FRt1/detection_loss‚1.6498327692772479 /loss/FRt3/detection_loss‚3.1337150988670497 0loss/SRqqt1/detection_loss‚0.895592894577063 .loss/FBt4/detection_loss‚3.107461074988047 0loss/FRt1/assignment_loss‚1.1964442818722827 0loss/FRt3/assignment_loss‚3.0283138064237742 1loss/SRqqt2/assignment_loss‚1.273472949298468 1loss/FBt2/assignment_loss‚0.33167818256102777 /loss/FBt1/detection_loss‚0.5234598454747863 _runtime‚8388.734745485 2loss/SRqqt1/assignment_loss‚0.8299342901660846 0loss/FBt4/assignment_loss‚0.3834845603754123 /loss/FRt2/detection_loss‚2.5432119551632137 1loss/SRqqt2/detection_loss‚1.4194160907946785 /loss/FBt3/detection_loss‚2.5221563935279847 2loss/SRqqt3/assignment_loss‚1.6348996154847555 1loss/FBt1/assignment_loss‚0.16558069305189724 1loss/SRqqt4/detection_loss‚1.5878410466015338 /loss/FBt2/detection_loss‚2.0240909915703993 %loss/total_loss‚40.24055009741805 trainer/global_step‚11049 /loss/FRt2/assignment_loss‚2.508428845140669 1loss/SRqqt4/assignment_loss‚1.313946454351147 0loss/FBt3/assignment_loss‚0.6560337394825183 epoch‚9 .loss/FRt4/detection_loss‚2.286000685982926 1loss/SRqqt3/detection_loss‚2.4429666493088007 % _timestamp‚1.7884367040300946e+09 _step‚910Ž鬵“€(‚(:ˆ ëÁåÔý¤Ž gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.691' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9382 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A ·3"(‚(:— ëÁåÔˆïî$ disk.nvme0n1.out‚50743.814453125# disk.nvme0c0n1.in‚ 1076.44921875 disk.rbd0.in‚0$ memory_percent‚94.19066905975342" proc.memory.rssMB‚ 2872.3515625 cpu‚25.365494799478$ disk./.usageGB‚1100.825626373291 disk.sda1.out‚0 network.recv‚ 2.862487e+06 disk.rbd1.out‚ 4.29296875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚338.875! disk.nvme0n1.in‚ 1076.44921875& disk.nvme0c0n1.out‚50743.814453125 disk.sda3.out‚338.875* proc.memory.percent‚17.531442642211914 disk.dm-0.in‚ 1076.44921875! disk.dm-0.out‚50743.814453125 disk.sda.out‚338.875 disk.sda1.in‚0 disk.sda3.in‚0.5 network.sent‚ 2.676549e+07 disk.sda.in‚0.5! disk.rbd2.out‚221.39111328125 disk.drbd1777.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 951.80078125) disk./.usagePercent‚73.87733144013372 disk.dm-1.in‚0.5 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0óW·“‚(‚(:ˆ úÁåÔ¾±“Ž gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚102.562' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@4wM6ƒ(‚(:« úÁåÔâ›ë disk.sda2.out‚0 disk.sda3.out‚ 339.26171875 disk.sda.out‚ 339.26171875 disk.rbd3.out‚0% disk./.usageGB‚1100.8256492614746 disk.dm-0.in‚ 1076.44921875 disk.sda1.in‚0* proc.memory.percent‚17.531442642211914) disk./.usagePercent‚73.87733297617893! disk.dm-0.out‚50744.115234375 disk.dm-1.out‚ 339.26171875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 951.67578125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.29296875& disk.nvme0c0n1.out‚50744.115234375 disk.sda1.out‚0 disk.dm-1.in‚0.5! disk.nvme0n1.in‚ 1076.44921875$ memory_percent‚94.19143199920654 cpu‚25.364934350872343$ disk.nvme0n1.out‚50744.115234375 network.recv‚ 2.868539e+06! disk.rbd2.out‚221.39111328125 disk.sda3.in‚0.5 disk.drbd1777.out‚0 disk.sda.in‚0.5" proc.memory.rssMB‚ 2872.3515625 network.sent‚ 2.6873243e+07# disk.nvme0c0n1.in‚ 1076.44921875”+Âþ2„(- _wandbruntime‚8424  _runtime‚8424ÞË¡xÇ…(Á & lr-AdamW/pg1‚4.606681249176397e-06 & lr-AdamW/pg2‚4.606681249176397e-06 trainer/global_step‚11099 % _timestamp‚1.7884367395908616e+09 _runtime‚ 8424.29560648 _step‚911ô¬ºŎ(Æ _runtime‚ 8424.29560648 _step‚911 & lr-AdamW/pg1‚4.606681249176397e-06 & lr-AdamW/pg2‚4.606681249176397e-06 trainer/global_step‚11099 % _timestamp‚1.7884367395908616e+09à\[´2‡(- _wandbruntime‚8424  _runtime‚8424˜…rTˆ(áO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161712198 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚646418270028288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94358270 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚11099 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚629463094234112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209755663 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53167824 eQDev­ÿ àÊ"iceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚705819384278528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161712200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9761422964736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147526092 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147526516 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚715580625970688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚655724029732864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9305777707520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚705819360338432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚646418252025344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚629463111273472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8424.296566549 _step‚912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9272168195418 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9271513757662 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209735625 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚715580807243264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚638735279468890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚646418270028288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚655724029732864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9761265632256 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9306434070016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209755663 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94358268 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040]γì'‰(æO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚646418252025344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚629463111273472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8424.296566549 _step‚912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9272168195418 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9271513757662 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209735625 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚715580807243264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚638735279468890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚646418270028288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚655724029732864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9761265632256 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9306434070016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209755663 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94358268 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161712198 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚646418270028288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94358270 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚11099 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚629463094234112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209755663 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53167824 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚705819384278528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161712200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9761422964736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147526092 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147526516 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚715580625970688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚655724029732864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9305777707520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚705819360338432b¨Ê2Š(- _wandbruntime‚8424  _runtime‚8424<´0Mò&‹(ìM EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147538737 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9762120887808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371480293 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9306597135360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚705881011076096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9307253511168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371480293 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209774143 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚629517998364672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚646474623067136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94366436 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53172741 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209754079 % _timestamp‚1.7884367402772954e+09 _runtime‚8424.982715989 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161726214 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161726216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚655781876578304 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371500359 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9306597135360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚705881035016192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161726214 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209774143 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚913 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371500359 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚646474605064192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161726216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚655781876578304 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371500359 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9306597135360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚705881035016192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161726214 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209774143 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚913 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371500359 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚646474605064192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147538737 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9762120887808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371480293 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9306597135360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚705881011076096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9307253511168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371480293 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209774143 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚629517998364672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚646474623067136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94366436 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53172741 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209754079 % _timestamp‚1.7884367402772954e+09 _runtime‚8424.982715989‘™i}Ð2(- _wandbruntime‚8424  _runtime‚8424cWÝ_Ž(Ù .loss/FBt2/detection_loss‚4.328350138664246 1loss/FBt1/assignment_loss‚0.08905164883065565 0loss/SRqqt3/detection_loss‚2.193173525854945 epoch‚9 _runtime‚8424.984154619 1loss/SRqqt2/assignment_loss‚1.212549055501688 .loss/FRt1/detection_loss‚1.478622965612694 /loss/FRt2/detection_loss‚2.1966066543872538 trainer/global_step‚11099 0loss/FRt2/assignment_loss‚2.7070040576733074 /loss/FBt1/detection_loss‚0.5188244908772917 /loss/FRt3/assignment_loss‚3.097089423239231 0loss/FRt4/assignment_loss‚3.1593624080220857 2loss/FBt2/assignment_loss‚0.018001484431351855 /loss/FRt3/detection_loss‚2.6343544140458106 /loss/FRt4/detection_loss‚2.2844138834625483 % _timestamp‚1.7884367402795148e+09 _step‚914 1loss/FBt3/assignment_loss‚0.24635692383162677 1loss/SRqqt1/detection_loss‚0.8531526650866152 0loss/SRqqt2/detection_loss‚1.695794565635815 &loss/total_loss‚41.731333813558635 0loss/FRt1/assignment_loss‚1.2140465723840814 2loss/SRqqt3/assignment_loss‚1.7161605334840715 2loss/SRqqt4/assignment_loss‚1.2577494920955763 1loss/SRqqt4/detection_loss‚2.3396541786690555 /loss/FBt3/detection_loss‚2.6487001478672028 /loss/FBt4/detection_loss‚2.4988122642040254 1loss/SRqqt1/assignment_loss‚0.904529830312986 1loss/FBt4/assignment_loss‚0.43897248938446864«LÄd(Þ 0loss/FRt2/assignment_loss‚2.7070040576733074 /loss/FBt1/detection_loss‚0.5188244908772917 /loss/FRt3/assignment_loss‚3.097089423239231 0loss/FRt4/assignment_loss‚3.1593624080220857 2loss/FBt2/assignment_loss‚0.018001484431351855 /loss/FRt3/detection_loss‚2.6343544140458106 /loss/FRt4/detection_loss‚2.2844138834625483 % _timestamp‚1.7884367402795148e+09 _step‚914 1loss/FBt3/assignment_loss‚0.24635692383162677 1loss/SRqqt1/detection_loss‚0.8531526650866152 0loss/SRqqt2/detection_loss‚1.695794565635815 &loss/total_loss‚41.731333813558635 0loss/FRt1/assignment_loss‚1.2140465723840814 2loss/SRqqt3/assignment_loss‚1.7161605334840715 2loss/SRqqt4/assignment_loss‚1.2577494920955763 1loss/SRqqt4/detection_loss‚2.3396541786690555 /loss/FBt3/detection_loss‚2.6487001478672028 /loss/FBt4/detection_loss‚2.4988122642040254 1loss/SRqqt1/assignment_loss‚0.904529830312986 1loss/FBt4/assignment_loss‚0.43897248938446864 .loss/FBt2/detection_loss‚4.328350138664246 1loss/FBt1/assignment_loss‚0.08905164883065565 0loss/SRqqt3/detection_loss‚2.193173525854945 epoch‚9 _runtime‚8424.984154619 1loss/SRqqt2/assignment_loss‚1.212549055501688 .loss/FRt1/detection_loss‚1.478622965612694 /loss/FRt2/detection_loss‚2.1966066543872538 trainer/global_step‚11099’sÆ°‘(‚(:† ‰ÂåÔÔŒˆŽ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.06' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>ÿ|í;‘(‚(:° ‰ÂåÔè‘ disk.rbd3.in‚0 disk.dm-0.in‚ 1076.44921875 network.sent‚ 2.6879558e+07% disk./.usageGB‚1100.8256301879883 disk.sda1.out‚0 disk.sda3.in‚0.5# disk.dm-0.out‚50843.55224609375* proc.memory.percent‚17.531442642211914 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1076.44921875 disk.rbd1.in‚0 disk.sda2.in‚0# memory_percent‚94.1918134689331( proc.memory.availableMB‚ 951.61328125 disk.sda.out‚ 340.12890625# disk.nvme0c0n1.in‚ 1076.44921875! disk.rbd2.out‚223.05908203125 disk.sda3.out‚ 340.12890625 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2872.3515625 disk.rbd1.out‚ 4.29296875 disk.sda.in‚0.5 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 cpu‚25.364372742643734 disk.drbd1777.in‚0 network.recv‚ 2.873119e+06) disk./.usagePercent‚73.87733169614125( disk.nvme0c0n1.out‚50843.55224609375 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0.5 disk.dm-1.out‚ 340.12890625& disk.nvme0n1.out‚50843.55224609375A@Éô’(‚(:’ ˜ÂåÔ¬åŽ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚89.372' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.874399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 2¦Ï5“(‚(:ª ˜ÂåÔ‹­¦‘) disk./.usagePercent‚73.87718244374881 disk.sda3.out‚ 340.640625 disk.drbd1777.out‚0 disk.dm-1.out‚ 340.640625! disk.nvme0n1.in‚ 1076.44921875 disk.rbd0.out‚0( proc.memory.availableMB‚ 951.36328125 network.recv‚ 2.879009e+06% disk./.usageGB‚1100.8234062194824# disk.dm-0.out‚50942.70654296875# disk.nvme0c0n1.in‚ 1076.44921875( disk.nvme0c0n1.out‚50942.70654296875 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 340.640625 disk.rbd1.in‚0 disk.rbd0.in‚0! disk.rbd2.out‚223.05908203125 proc.cpu.threads‚17 network.sent‚ 2.6987434e+07 disk.sda3.in‚0.5 disk.sda.in‚0.5 cpu‚25.36381609690365 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚50942.70654296875 disk.sda2.out‚0* proc.memory.percent‚17.531442642211914 disk.dm-0.in‚ 1076.44921875" proc.memory.rssMB‚ 2872.3515625 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0.5 disk.rbd1.out‚ 4.29296875 disk.sda2.in‚0$ memory_percent‚94.19333934783936f;ˆ2”(- _wandbruntime‚8459  _runtime‚8459««IBÊ•(Ä _step‚915 ' lr-AdamW/pg1‚4.1563481190595025e-06 ' lr-AdamW/pg2‚4.1563481190595025e-06 trainer/global_step‚11149 % _timestamp‚1.7884367751860518e+09 _runtime‚8459.890801342ÃXN1Ï–(É ' lr-AdamW/pg1‚4.1563481190595025e-06 ' lr-AdamW/pg2‚4.1563481190595025e-06 trainer/global_step‚11149 % _timestamp‚1.7884367751860518e+09 _runtime‚8459.890801342 _step‚915“ê¢Â2—(- _wandbruntime‚8459  _runtime‚8459¯‘Oè'˜(âO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9347419875840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53392215 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649235921970688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9347419875840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162413000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚718706250791424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9804353569792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚708901921161728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚658583341846528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210681715 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9346763513344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649235903967744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 373094715 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9312349097070 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚632208317801472 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649235903967744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162412998 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210681715 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚708901897221632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9313003534826 % _timestamp‚1.7884367751868098e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658582667481088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148158452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94766668 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53391784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚718706434161152 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚916 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649235921970688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚658583341846528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚658582667481088&å„2&™(çO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148158452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94766668 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53391784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚718706434161152 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚916 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649235921970688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚658583341846528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚658582667481088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9347419875840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53392215 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649235921970688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9347419875840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162413000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚718706250791424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9804353569792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚708901921161728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚658583341846528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210681715 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9346763513344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649235903967744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 373094715 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9312349097070 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚632208317801472 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649235903967744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162412998 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210681715 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚708901897221632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9313003534826 % _timestamp‚1.7884367751868098e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658582667481088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128”d)ÐO’š(‚(:‡ §ÂåÔ–¡ç gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚88.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZÜ4›(‚(:© §ÂåÔÿÍÛ disk.dm-0.in‚ 1076.44921875" proc.memory.rssMB‚ 2872.3515625" disk.dm-0.out‚51042.1259765625 disk.sda1.in‚0 disk.dm-1.out‚ 341.6640625 network.recv‚ 2.883718e+06$ disk./.usageGB‚1100.823387145996 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0.5 disk.rbd1.in‚0! disk.rbd2.out‚223.19970703125 disk.sda.in‚0.5 disk.dm-1.in‚0.5* proc.memory.percent‚17.531442642211914 proc.cpu.threads‚17' disk.nvme0c0n1.out‚51042.1259765625 disk.rbd0.out‚0 network.sent‚ 2.699376e+07) disk./.usagePercent‚73.87718116371113 disk.rbd1.out‚ 4.29296875 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 341.6640625 cpu‚25.363265052208714 disk.rbd2.in‚0 disk.sda3.out‚ 341.6640625 disk.rbd3.out‚0% disk.nvme0n1.out‚51042.1259765625# disk.nvme0c0n1.in‚ 1076.44921875 disk.sda2.out‚0! disk.nvme0n1.in‚ 1076.44921875$ memory_percent‚94.19333934783936( proc.memory.availableMB‚ 951.36328125 disk.drbd1777.in‚0 disk.sda1.out‚0tø2œ(- _wandbruntime‚8460  _runtime‚8460ó1Öžò&(ìM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚649292257006592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94774838 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚708963547959296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210680131 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚422 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9805379949056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚718768772686336 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210680131 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚641577041902030 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9348239316992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚718768951848448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚632263204892672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9313165532034 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚649292275009536 trainer/global_step‚11149 _step‚917 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚708963571899392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 373127211 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649292275009536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚658639839947776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162427016 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9313819969998 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 373127211 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210700195 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148171539 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚632263221932032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚658640514326528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884367758429697e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9348239316992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 373107145 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚641576370424706 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚658640514326528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 373107145 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162427016 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649292257006592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9347582941184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162427014 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9347582941184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚8460.548231924 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148171117 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210700195 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162427014:Íë÷&ž(ñM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210700195 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148171539 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚632263221932032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚658640514326528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884367758429697e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9348239316992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 373107145 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚641576370424706 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚658640514326528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 373107145 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162427016 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649292257006592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9347582941184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162427014 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9347582941184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚8460.548231924 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148171117 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210700195 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162427014 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚649292257006592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94774838 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚708963547959296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210680131 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚422 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9805379949056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚718768772686336 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210680131 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚641577041902030 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9348239316992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚718768951848448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚632263204892672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9313165532034 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚649292275009536 trainer/global_step‚11149 _step‚917 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚708963571899392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 373127211 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649292275009536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚658639839947776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162427016 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9313819969998 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 373127211•|²2Ÿ(- _wandbruntime‚8460  _runtime‚84607GäòZ (Ô epoch‚9 2loss/SRqqt2/assignment_loss‚1.4269993178270481 /loss/FRt4/detection_loss‚2.1362253268559774 /loss/FBt1/detection_loss‚0.5743929750974437 .loss/FBt2/detection_loss‚2.484487843513489 trainer/global_step‚11149 .loss/FRt3/assignment_loss‚3.09336001597918 1loss/FBt1/assignment_loss‚0.25017660951675064 0loss/FBt4/assignment_loss‚0.6000890698791905 /loss/FRt1/detection_loss‚1.5373745591000274 .loss/FRt2/detection_loss‚2.280219738213521 % _timestamp‚1.7884367758446522e+09 2loss/SRqqt3/assignment_loss‚1.7470778040587902 0loss/FBt2/assignment_loss‚0.1803240017456119 _runtime‚8460.549704529 _step‚918 0loss/FRt1/assignment_loss‚0.9954603888370372 1loss/SRqqt1/detection_loss‚0.8889931143459046 1loss/SRqqt4/detection_loss‚1.5431538937183527 0loss/SRqqt3/detection_loss‚2.397893648255955 /loss/FBt4/detection_loss‚1.9985491145740857 /loss/FRt2/assignment_loss‚2.353615569618513 0loss/FRt4/assignment_loss‚2.8129750728607177 2loss/SRqqt4/assignment_loss‚1.3013262573916178 1loss/FBt3/assignment_loss‚0.13057511332832897 %loss/total_loss‚39.28807219395924 1loss/SRqqt1/assignment_loss‚0.566804126575307 /loss/FRt3/detection_loss‚3.1702828338513007 1loss/SRqqt2/detection_loss‚1.8177969604730606 .loss/FBt3/detection_loss‚2.999918838342031¼[¢š_¡(Ù 0loss/FBt4/assignment_loss‚0.6000890698791905 /loss/FRt1/detection_loss‚1.5373745591000274 .loss/FRt2/detection_loss‚2.280219738213521 % _timestamp‚1.7884367758446522e+09 2loss/SRqqt3/assignment_loss‚1.7470778040587902 0loss/FBt2/assignment_loss‚0.1803240017456119 _runtime‚8460.549704529 _step‚918 0loss/FRt1/assignment_loss‚0.9954603888370372 1loss/SRqqt1/detection_loss‚0.8889931143459046 1loss/SRqqt4/detection_loss‚1.5431538937183527 0loss/SRqqt3/detection_loss‚2.397893648255955 /loss/FBt4/detection_loss‚1.9985491145740857 /loss/FRt2/assignment_loss‚2.353615569618513 0loss/FRt4/assignment_loss‚2.8129750728607177 2loss/SRqqt4/assignment_loss‚1.3013262573916178 1loss/FBt3/assignment_loss‚0.13057511332832897 %loss/total_loss‚39.28807219395924 1loss/SRqqt1/assignment_loss‚0.566804126575307 /loss/FRt3/detection_loss‚3.1702828338513007 1loss/SRqqt2/detection_loss‚1.8177969604730606 .loss/FBt3/detection_loss‚2.999918838342031 epoch‚9 2loss/SRqqt2/assignment_loss‚1.4269993178270481 /loss/FRt4/detection_loss‚2.1362253268559774 /loss/FBt1/detection_loss‚0.5743929750974437 .loss/FBt2/detection_loss‚2.484487843513489 trainer/global_step‚11149 .loss/FRt3/assignment_loss‚3.09336001597918 1loss/FBt1/assignment_loss‚0.25017660951675064–²5(M“¢(‚(:ˆ ¶ÂåÔ‰ß gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.334' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªJì;0£(‚(:¥ ¶ÂåÔ™¨¾' proc.memory.availableMB‚ 951.5703125 network.recv‚ 2.889642e+06% disk.nvme0c0n1.out‚51042.53515625 disk.sda2.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.in‚0$ memory_percent‚94.19207572937012* proc.memory.percent‚17.531442642211914 cpu‚25.3626842981008) disk./.usagePercent‚73.87718346777893 disk.rbd0.out‚0 disk.sda3.in‚0.5 disk.sda3.out‚ 342.29296875 disk.drbd1777.in‚0 disk.dm-0.out‚51042.53515625! disk.nvme0n1.in‚ 1076.44921875! disk.rbd2.out‚223.19970703125 disk.rbd3.in‚0% disk./.usageGB‚1100.8234214782715 disk.sda.out‚ 342.29296875 disk.drbd1777.out‚0 disk.dm-0.in‚ 1076.44921875 disk.dm-1.in‚0.5 disk.sda2.out‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2872.3515625 disk.sda1.in‚0 disk.dm-1.out‚ 342.29296875# disk.nvme0n1.out‚51042.53515625 disk.rbd1.out‚ 4.29296875 network.sent‚ 2.7090323e+07 disk.sda.in‚0.5 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1076.44921875 Bœ·ž¤(‚(:“ ÅÂåÔã±ú gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.558' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.911599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿&z9¥(‚(:® ÅÂåÔ·¬‘) disk./.usagePercent‚73.87718346777893 disk.sda3.out‚ 343.08984375 disk.sda.in‚0.5" proc.memory.rssMB‚ 2872.3515625 proc.cpu.threads‚17 disk.dm-0.in‚ 1076.44921875 disk.sda1.in‚0! disk.nvme0n1.in‚ 1076.44921875 cpu‚25.362130265375757% disk./.usageGB‚1100.8234214782715 disk.sda2.out‚0& proc.memory.availableMB‚ 951.328125* proc.memory.percent‚17.531442642211914 disk.rbd2.out‚223.3642578125 disk.rbd3.out‚0# disk.dm-0.out‚51141.86572265625 disk.rbd1.in‚0 network.recv‚ 2.895854e+06 disk.rbd3.in‚0 disk.sda.out‚ 343.08984375 disk.rbd1.out‚ 4.29296875# disk.nvme0c0n1.in‚ 1076.44921875( disk.nvme0c0n1.out‚51141.86572265625 network.sent‚ 2.7188416e+07 disk.drbd1777.in‚0& disk.nvme0n1.out‚51141.86572265625 disk.rbd0.out‚0 disk.sda3.in‚0.5 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0.5 disk.dm-1.out‚ 343.08984375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0$ memory_percent‚94.19355392456055Æ”u~2¦(- _wandbruntime‚8495  _runtime‚8495¤2q1ɧ(à _runtime‚8495.960799475 _step‚919 ' lr-AdamW/pg1‚3.7287627388488413e-06 ' lr-AdamW/pg2‚3.7287627388488413e-06 trainer/global_step‚11199 $ _timestamp‚1.788436811255929e+09$N¼©Î¨(È trainer/global_step‚11199 $ _timestamp‚1.788436811255929e+09 _runtime‚8495.960799475 _step‚919 ' lr-AdamW/pg1‚3.7287627388488413e-06 ' lr-AdamW/pg2‚3.7287627388488413e-06—#8øÅ2©(- _wandbruntime‚8495  _runtime‚84953PQNfª(âO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211607767 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53616536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374721567 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163113798 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 % _timestamp‚1.7884368112567253e+09 _step‚920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚661441305229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95175070 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711984458044928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_bab@攂tch_start/allocated_bytes.all.allocated‚661441979594752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652053555910144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚652053555910144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634953524329472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148791606 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9847350454784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚652053573913088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211587729 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚661441979594752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148791181 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚661441305229312 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211607767 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53616113 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9353838874234 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9387749319168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9847188928000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚11199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95175068 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚711984434104832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9353184436478 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚721831808499712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚644307363203706 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚661441305229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95175070 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711984458044928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚661441979594752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652053555910144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚652053555910144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634953524329472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148791606 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9847350454784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚652053573913088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211587729 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚661441979594752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148791181 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚661441305229312 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211607767 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53616113 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9353838874234 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9387749319168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9847188928000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚11199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95175068 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚711984434104832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9353184436478 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚721831808499712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚644307363203706 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889˜Ͻó,2¬(- _wandbruntime‚8496  _runtime‚8496,µa¶ò&­(ìM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95183236 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚644363083769438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9354655309406 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚652109908948992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9389225122816 _step‚921 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211606183 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚712046108782592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163127814 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚652109908948992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95183238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚712046084842496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148804225 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚635008411420672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚652109926951936 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163127814 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374733997 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚8496.635603958 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211606183 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9389225122816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374754063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚434 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148803791 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211626247 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9388568747008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374754063 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚712046108782592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163127814 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚652109908948992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95183238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚712046084842496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148804225 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚635008411420672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚652109926951936 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163127814 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374733997 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚8496.635603958 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211606183 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9389225122816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374754063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚434 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148803791 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211626247 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9388568747008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374754063 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664299942977536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚715066970988032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95583470 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163814598 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53842713 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11249 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚647092733593998 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9394019775886 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚654871207852544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95583468 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚637698730857472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212513781 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚654871207852544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212513781 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163814598 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149426183 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9890358272000 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 $ _timestamp‚1.788436847164189e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212533819 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9429391487488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9394674213642 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163814600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚647093405071114à,³ì'»(æO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9394674213642 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163814600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚647093405071114 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚724957163538944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163814600 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664299942977536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚715066970988032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95583470 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163814598 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53842713 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11249 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚647092733593998 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9394019775886 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚654871207852544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95583468 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚637698730857472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212513781 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚654871207852544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212513781 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163814598 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149426183 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9890358272000 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 $ _timestamp‚1.788436847164189e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212533819 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9429391487488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680œ„ ÿÐ2¼(- _wandbruntime‚8532  _runtime‚8532AŽj ½(ìM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚664357789822976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212532235 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9395490648814 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163828614 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9429554552832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95591638 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9891059704320 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376380915 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚11249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95591636 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149438772 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚637753617948672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149438336 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚664357789822976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚654927560891392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9430210928640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚725019681430016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9430210928640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163828616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53846700 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚715128645665792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚654927560891392 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163828614 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚654927578894336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376380915 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53847134 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376360849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚664357115444224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376360849 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9394836210850 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212552299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884368478480184e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212552299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚654927578894336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9891219120640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _step‚925¿¾…÷&¾(ñM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884368478480184e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212552299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚654927578894336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9891219120640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _step‚925 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚664357789822976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212532235 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9395490648814 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163828614 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9429554552832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95591638 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9891059704320 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376380915 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚11249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95591636 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149438772 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚637753617948672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149438336 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚664357789822976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚654927560891392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9430210928640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚725019681430016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9430210928640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163828616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53846700 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚715128645665792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚654927560891392 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163828614 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚654927578894336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376380915 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53847134 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376360849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚664357115444224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376360849 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9394836210850 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212552299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414ð×fš2¿(- _wandbruntime‚8532  _runtime‚85321Ü'^À(Ø 0loss/FRt3/assignment_loss‚2.6934536695480347 1loss/SRqqt3/assignment_loss‚1.727851318685632 .loss/FRt4/detection_loss‚2.240706008214217 %loss/total_loss‚41.51875217299764 epoch‚9 _step‚926 /loss/FRt4/assignment_loss‚3.262244760990143 2loss/SRqqt2/assignment_loss‚1.4915773547612703 1loss/FBt2/assignment_loss‚0.18875050875612942 trainer/global_step‚11249 0loss/FRt2/assignment_loss‚2.5674721961436062 0loss/FRt1/assignment_loss‚1.6324191907557046 1loss/SRqqt1/detection_loss‚1.2212910642502484 0loss/SRqqt3/detection_loss‚2.243477463722229 /loss/FBt1/detection_loss‚0.6862342197604492 % _timestamp‚1.7884368478496752e+09 2loss/SRqqt1/assignment_loss‚1.0400838351687762 1loss/FBt4/assignment_loss‚0.25282298466966796 /loss/FRt1/detection_loss‚1.9185287592160265 _runtime‚8532.554305993 /loss/FRt2/detection_loss‚2.5192249657019326 /loss/FRt3/detection_loss‚2.6664440302472365 1loss/SRqqt2/detection_loss‚1.5079081954864355 0loss/SRqqt4/detection_loss‚1.599158638715744 /loss/FBt2/detection_loss‚2.5550439804792404 /loss/FBt3/detection_loss‚2.9921697080135345 1loss/SRqqt4/assignment_loss‚1.681552399881184 2loss/FBt3/assignment_loss‚0.020064522745087743 1loss/FBt1/assignment_loss‚0.09307319073287236 .loss/FBt4/detection_loss‚2.717199206352234W}pcÁ(Ý % _timestamp‚1.7884368478496752e+09 2loss/SRqqt1/assignment_loss‚1.0400838351687762 1loss/FBt4/assignment_loss‚0.25282298466966796 /loss/FRt1/detection_loss‚1.9185287592160265 _runtime‚8532.554305993 /loss/FRt2/detection_loss‚2.5192249657019326 /loss/FRt3/detection_loss‚2.6664440302472365 1loss/SRqqt2/detection_loss‚1.5079081954864355 0loss/SRqqt4/detection_loss‚1.599158638715744 /loss/FBt2/detection_loss‚2.5550439804792404 /loss/FBt3/detection_loss‚2.9921697080135345 1loss/SRqqt4/assignment_loss‚1.681552399881184 2loss/FBt3/assignment_loss‚0.020064522745087743 1loss/FBt1/assignment_loss‚0.09307319073287236 .loss/FBt4/detection_loss‚2.717199206352234 0loss/FRt3/assignment_loss‚2.6934536695480347 1loss/SRqqt3/assignment_loss‚1.727851318685632 .loss/FRt4/detection_loss‚2.240706008214217 %loss/total_loss‚41.51875217299764 epoch‚9 _step‚926 /loss/FRt4/assignment_loss‚3.262244760990143 2loss/SRqqt2/assignment_loss‚1.4915773547612703 1loss/FBt2/assignment_loss‚0.18875050875612942 trainer/global_step‚11249 0loss/FRt2/assignment_loss‚2.5674721961436062 0loss/FRt1/assignment_loss‚1.6324191907557046 1loss/SRqqt1/detection_loss‚1.2212910642502484 0loss/SRqqt3/detection_loss‚2.243477463722229 /loss/FBt1/detection_loss‚0.6862342197604492žºh\«›Â(‚(: òÂåÔøøñ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.29' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.458000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y«Õ-Ã(‚(:¢ òÂåÔÒ£‘! disk.nvme0c0n1.in‚ 1076.453125 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.sda.in‚0.5 disk.rbd1.out‚ 4.29296875* proc.memory.percent‚17.531442642211914( disk.nvme0c0n1.out‚51343.12353515625 disk.rbd0.out‚0 disk.dm-0.in‚ 1076.453125 disk.sda1.in‚0 disk.dm-1.in‚0.5" proc.memory.rssMB‚ 2872.3515625 cpu‚25.360517542229417 disk.nvme0n1.in‚ 1076.453125& disk.nvme0n1.out‚51343.12353515625 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0' proc.memory.availableMB‚ 951.4765625 network.sent‚ 2.7306843e+07) disk./.usagePercent‚73.87730916747824 disk.sda2.in‚0 disk.rbd2.out‚225.0791015625 disk.drbd1777.out‚0# disk.dm-0.out‚51343.12353515625$ memory_percent‚94.19264793395996 disk.sda3.out‚ 345.109375 disk.rbd3.in‚0 disk.sda1.out‚0$ disk./.usageGB‚1100.825294494629 disk.sda3.in‚0.5 disk.sda.out‚ 345.109375 disk.dm-1.out‚ 345.109375 proc.cpu.threads‚17 network.recv‚ 2.909681e+06çF½/žÄ(‚(:“ ÃåÔ–öÿ gpu.0.gpu‚33.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.908' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.981599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01¬qð5Å(‚(:ª ÃåÔдª‘$ memory_percent‚94.19264793395996 network.recv‚ 2.916526e+06 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda.in‚0.5" proc.memory.rssMB‚ 2872.3515625 cpu‚25.359944618469488 disk.dm-0.in‚ 1076.453125 disk.sda.out‚ 345.76953125 disk.dm-1.in‚0.5 disk.drbd1777.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 network.sent‚ 2.7415708e+07) disk./.usagePercent‚73.87730942348578# disk.dm-0.out‚51442.72509765625 disk.dm-1.out‚ 345.76953125 disk.rbd1.out‚ 4.37890625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 1076.453125 disk.sda2.in‚0' proc.memory.availableMB‚ 951.4765625* proc.memory.percent‚17.531442642211914 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 345.76953125( disk.nvme0c0n1.out‚51442.72509765625 disk.sda2.out‚0! disk.rbd2.out‚225.24365234375 disk.sda3.in‚0.5% disk./.usageGB‚1100.8252983093262 disk.sda1.out‚0& disk.nvme0n1.out‚51442.72509765625 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 1076.453125`ÝuM’Æ(‚(:‡ ÃåÔàÁø gpu.0.gpu‚44.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚94.572' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9144 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ich0Ç(‚(:¥ ÃåÔ¨§% disk./.usageGB‚1100.8230667114258 disk.drbd1777.out‚0 disk.rbd3.in‚0! disk.dm-0.out‚51443.041015625$ disk.nvme0n1.out‚51443.041015625 disk.rbd1.out‚ 4.37890625* proc.memory.percent‚17.531442642211914 disk.sda.out‚ 346.27734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 346.27734375 disk.dm-0.in‚ 1076.453125 disk.dm-1.in‚0.5$ memory_percent‚94.19267177581787 proc.cpu.threads‚17 disk.sda3.in‚0.5 disk.nvme0n1.in‚ 1076.453125! disk.rbd2.out‚225.24365234375" proc.memory.rssMB‚ 2872.3515625 disk.rbd3.out‚0 disk.sda.in‚0.5 disk.sda2.in‚0 disk.rbd2.in‚0 cpu‚25.359438816734297 network.sent‚ 2.7421118e+07 network.recv‚ 2.920588e+06 disk.drbd1777.in‚0 disk.dm-1.out‚ 346.27734375! disk.nvme0c0n1.in‚ 1076.453125& disk.nvme0c0n1.out‚51443.041015625 disk.sda2.out‚0) disk./.usagePercent‚73.87715965907825 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 951.47265625 XeV2È(- _wandbruntime‚8567  _runtime‚8567é8vçÈÉ( & lr-AdamW/pg2‚2.942186901588607e-06 trainer/global_step‚11299 % _timestamp‚1.7884368831911485e+09 _runtime‚8567.895888538 _step‚927 & lr-AdamW/pg1‚2.942186901588607e-06•8À×ÍÊ(Ç & lr-AdamW/pg1‚2.942186901588607e-06 & lr-AdamW/pg2‚2.942186901588607e-06 trainer/global_step‚11299 % _timestamp‚1.7884368831911485e+09 _runtime‚8567.895888538 _step‚927Ÿ~/ü2Ë(- _wandbruntime‚8567  _runtime‚8567‹‹›çÈÌ(âO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164515398 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚657688877797888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164515398 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚649879446938522 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9435509553050 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚667159255091200 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚667159255091200 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚667158580725760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164515400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚718149531811328 _runtime‚8567.896843668 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚667158580725760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚657688877797888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9470377293312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95991870 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164515400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95991868 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884368831918597e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 377955231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚728082849868800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚84Cæ ~ 4 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9434855115294 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213439833 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54068147 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9469720930816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚640443920346112 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚640443937385472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9469720930816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213459871 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚657688859794944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150060457 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9933148142080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚649878775461406 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚718149531811328 _runtime‚8567.896843668 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚667158580725760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚657688877797888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9470377293312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95991870 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164515400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95991868 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884368831918597e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 377955231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚728082849868800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9434855115294 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213439833 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54068147 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9469720930816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚640443920346112 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚640443937385472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9469720930816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213459871 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚657688859794944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150060457 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9933148142080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚649878775461406 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164515398 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚649879446938522 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9435509553050 RÛw/2Î(- _wandbruntime‚8568  _runtime‚8568\P:Hñ&Ï(ëM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164529416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9470540358656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9934014410752 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8568.557255225 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657745212833792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 377987701 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚657745230836736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚667215753192448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150073074 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378007767 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657745230836736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213458287 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚728145173019648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54073036 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚667216427571200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164529414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9471196734464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96000036 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9436325988222 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213458287 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚728145358473216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9470540358656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378007767 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚640498841516032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚929 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚640498824476672 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9435671550258 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150072631 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚718211182548992 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164529416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9471196734464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96000038 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚657745212833792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54072595 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚649934496026930 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213478351I]u”­&Ð(ðM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378007767 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657745230836736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213458287 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚728145173019648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54073036 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚667216427571200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164529414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9471196734464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96000036 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9436325988222 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213458287 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚728145358473216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9470540358656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378007767 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚640498841516032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚929 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚640498824476672 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9435671550258 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150072631 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚718211182548992 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164529416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9471196734464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96000038 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚657745212833792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54072595 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚649934496026930 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213478351 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164529416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9470540358656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9934014410752 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8568.557255225 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657745212833792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 377987701 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚657745230836736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚667215753192448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150073074 E`@ªÝI?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0¡Ê«E 2Ñ(- _wandbruntime‚8568  _runtime‚8568 ߌÝ[Ò(Õ .loss/FBt3/detection_loss‚3.049606204032898 2loss/SRqqt4/assignment_loss‚1.4208957359533418 1loss/FBt2/assignment_loss‚0.03108398775657406 1loss/FBt4/assignment_loss‚0.10158805924907534 _runtime‚ 8568.558909 0loss/FRt1/assignment_loss‚1.1249851962415183 2loss/SRqqt1/assignment_loss‚0.7963131995844271 .loss/FBt2/detection_loss‚2.509285882115364 2loss/FBt3/assignment_loss‚0.007098162895971427 /loss/FRt1/detection_loss‚1.8146377174047434 0loss/FRt2/assignment_loss‚2.9047951017107283 1loss/SRqqt1/detection_loss‚0.8077447848284945 epoch‚9 1loss/SRqqt2/assignment_loss‚1.219059704365779 1loss/SRqqt3/assignment_loss‚1.808400927058288 0loss/SRqqt3/detection_loss‚2.193758923666818 %loss/total_loss‚38.98048056676718 .loss/FRt2/detection_loss‚2.521205710513251 /loss/FBt1/detection_loss‚0.5577959595450974 0loss/FRt3/assignment_loss‚2.6620094458262127 1loss/FBt1/assignment_loss‚0.10278629155659984 1loss/SRqqt4/detection_loss‚1.3906265085393732 /loss/FBt4/detection_loss‚2.3659766382641263 trainer/global_step‚11299 _step‚930 % _timestamp‚1.7884368838542082e+09 0loss/FRt4/assignment_loss‚3.0147960123262907 .loss/FRt3/detection_loss‚2.921083704630534 /loss/FRt4/detection_loss‚1.9231225001185168 1loss/SRqqt2/detection_loss‚1.7318242085831506’Ðé^`Ó(Ú 1loss/SRqqt2/assignment_loss‚1.219059704365779 1loss/SRqqt3/assignment_loss‚1.808400927058288 0loss/SRqqt3/detection_loss‚2.193758923666818 %loss/total_loss‚38.98048056676718 .loss/FRt2/detection_loss‚2.521205710513251 /loss/FBt1/detection_loss‚0.5577959595450974 0loss/FRt3/assignment_loss‚2.6620094458262127 1loss/FBt1/assignment_loss‚0.10278629155659984 1loss/SRqqt4/detection_loss‚1.3906265085393732 /loss/FBt4/detection_loss‚2.3659766382641263 trainer/global_step‚11299 _step‚930 % _timestamp‚1.7884368838542082e+09 0loss/FRt4/assignment_loss‚3.0147960123262907 .loss/FRt3/detection_loss‚2.921083704630534 /loss/FRt4/detection_loss‚1.9231225001185168 1loss/SRqqt2/detection_loss‚1.7318242085831506 .loss/FBt3/detection_loss‚3.049606204032898 2loss/SRqqt4/assignment_loss‚1.4208957359533418 1loss/FBt2/assignment_loss‚0.03108398775657406 1loss/FBt4/assignment_loss‚0.10158805924907534 _runtime‚ 8568.558909 0loss/FRt1/assignment_loss‚1.1249851962415183 2loss/SRqqt1/assignment_loss‚0.7963131995844271 .loss/FBt2/detection_loss‚2.509285882115364 2loss/FBt3/assignment_loss‚0.007098162895971427 /loss/FRt1/detection_loss‚1.8146377174047434 0loss/FRt2/assignment_loss‚2.9047951017107283 1loss/SRqqt1/detection_loss‚0.8077447848284945 epoch‚9¢àÖ|ò“Ô(‚(:ˆ ŸÃåÔÛÊÞ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.357' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6714 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù‘Ù(Õ(‚(: ŸÃåÔ›€ ! disk.dm-0.out‚51542.240234375 disk.sda1.in‚0 disk.dm-1.out‚ 346.8828125 disk.rbd1.in‚0 disk.dm-1.in‚0.5 disk.nvme0n1.in‚ 1076.453125 disk.sda2.in‚0 network.recv‚ 2.925271e+06% disk./.usageGB‚1100.8230476379395 disk.dm-0.in‚ 1076.453125 disk.rbd1.out‚ 4.44140625& disk.nvme0c0n1.out‚51542.240234375 cpu‚25.358899824931 network.sent‚ 2.7427392e+07 disk.sda.out‚ 346.8828125 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0$ disk.nvme0n1.out‚51542.240234375! disk.nvme0c0n1.in‚ 1076.453125 disk.sda2.out‚0$ memory_percent‚94.19584274291992 disk.rbd3.out‚0 disk.sda.in‚0.5 disk.rbd0.in‚0 disk.sda3.in‚0.5& proc.memory.availableMB‚ 950.953125 disk.drbd1777.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2872.3515625* proc.memory.percent‚17.531442642211914 disk.rbd2.in‚0! disk.rbd2.out‚225.38427734375 disk.sda3.out‚ 346.8828125) disk./.usagePercent‚73.87715837904058 disk.rbd3.in‚0’eøç“Ö(‚(:ˆ ®ÃåÔ˜·ŠŽ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.086' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"°0×(‚(:¥ ®Ãåԛ߂ disk.drbd1777.in‚0 disk.sda.in‚0.5 disk.sda.out‚ 347.546875 disk.dm-1.out‚ 347.546875! disk.nvme0c0n1.in‚ 1076.453125! disk.rbd2.out‚226.95849609375 disk.sda3.out‚ 347.546875 disk.dm-1.in‚0.5 disk.rbd1.in‚0( disk.nvme0c0n1.out‚51641.46728515625 disk.sda2.out‚0( proc.memory.availableMB‚ 951.16796875 network.sent‚ 2.7535156e+07 network.recv‚ 2.931563e+06 disk.sda3.in‚0.5 disk.rbd1.out‚ 4.44140625 disk.rbd0.out‚0* proc.memory.percent‚17.531442642211914 disk.rbd3.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚51641.46728515625 disk.rbd0.in‚0$ memory_percent‚94.19453144073486 disk.dm-0.in‚ 1076.453125 disk.sda1.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2872.3515625 cpu‚25.358334737318597) disk./.usagePercent‚73.87715837904058% disk./.usageGB‚1100.8230476379395 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚51641.46728515625 disk.nvme0n1.in‚ 1076.453125 proc.cpu.threads‚17n2Ø(- _wandbruntime‚8603  _runtime‚8603ü±­ÊÙ(Ä _runtime‚8603.679308034 _step‚931 ' lr-AdamW/pg1‚2.5833603657645936e-06 ' lr-AdamW/pg2‚2.5833603657645936e-06 trainer/global_step‚11349 % _timestamp‚1.7884369189745972e+09CTzÏÚ(É _step‚931 ' lr-AdamW/pg1‚2.5833603657645936e-06 ' lr-AdamW/pg2‚2.5833603657645936e-06 trainer/global_step‚11349 % _timestamp‚1.7884369189745972e+09 _runtime‚8603.679308034£mšV2Û(- _wandbruntime‚8603  _runtime‚8603è£ç'Ü(áO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚721232068694528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚652664817328814 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚660506529740288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 214385923 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150694576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9976243071488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚643189126874112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9510706736640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884369189753864e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165216200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9475690454702 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54294308 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚11349 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 214365885 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165216198 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚660506529740288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54294743 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚652665488805930 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9976398306816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚731208287825920 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96400270 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670017892839424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9510706736640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚670017892839424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚932 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165216198 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚ 8603.68029817 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9511363099136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 379582083 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚660506511737344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670017218473984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9511363099136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚731208467001344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150695013 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚660506511737344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚660506529740288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 214385923 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150694576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9976243071488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚643189126874112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9510706736640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884369189753864e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165216200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9475690454702 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54294308 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚11349 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 214365885 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165216198 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚660506529740288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54294743 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚652665488805930 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9976398306816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚731208287825920 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96400270 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670017892839424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9510706736640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚670017892839424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚932 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165216198 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚ 8603.68029817 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9511363099136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 379582083 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚660506511737344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670017218473984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9511363099136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚731208467001344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150695013 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚660506511737344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96408436 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9511526164480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9977267993600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150707344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚721293719432192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884369196388552e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165230216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9512182540288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚670074390940672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96408438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54299371 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚652721209371662 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚660562882779136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚643244048044032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚660562864776192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379634619 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9511526164480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚731270987425792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚643244031004672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165230214 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚660562864776192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54298908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9977102285824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150707809 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚670075065319424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚721293695492096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚660562882779136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165230216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9476506889666 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚670075065319424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 379634619 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214404403 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚652720537894338 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 379614553 _step‚933š,·r÷&à(ñM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165230214 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚660562864776192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54298908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9977102285824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150707809 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚670075065319424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚721293695492096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚660562882779136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165230216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9476506889666 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚670075065319424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 379634619 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214404403 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚652720537894338 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 379614553 _step‚933 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9477161327630 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚11349 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214384339 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚731270797777920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚8604.344315452 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96408436 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9511526164480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9977267993600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150707344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚721293719432192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884369196388552e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165230216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9512182540288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚670074390940672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96408438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54299371 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚652721209371662 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚660562882779136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚643244048044032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚660562864776192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379634619 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9511526164480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚731270987425792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚643244031004672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414¥• -?2á(- _wandbruntime‚8604  _runtime‚8604ð"c±Zâ(Ô -loss/FBt2/detection_loss‚3.38371342420578 /loss/FBt3/detection_loss‚2.8464234200390903 % _timestamp‚1.7884369196408134e+09 0loss/FRt1/assignment_loss‚0.9433319243517789 epoch‚9 0loss/FRt4/assignment_loss‚3.0616043051083883 1loss/FBt1/assignment_loss‚0.18257178190964876 0loss/SRqqt1/detection_loss‚1.008178663691358 /loss/SRqqt4/detection_loss‚1.78962832562766 2loss/SRqqt4/assignment_loss‚0.9708299974225602 .loss/FRt4/detection_loss‚2.235518068406317 _runtime‚8604.345456913 0loss/FRt2/assignment_loss‚2.3300637722015383 2loss/SRqqt1/assignment_loss‚0.9510899087606323 1loss/SRqqt3/assignment_loss‚1.549165670518522 1loss/FBt3/assignment_loss‚0.33775490999956176 0loss/SRqqt2/detection_loss‚1.838340866799448 /loss/FBt1/detection_loss‚0.5785418762492089 _step‚934 trainer/global_step‚11349 2loss/SRqqt2/assignment_loss‚1.6607755093890078 /loss/FRt2/detection_loss‚2.2510289831293955 /loss/FRt3/detection_loss‚2.8560515353756566 %loss/total_loss‚40.53368811937854 /loss/FRt3/assignment_loss‚2.738526607713392 0loss/SRqqt3/detection_loss‚2.500678691599104 .loss/FBt4/detection_loss‚3.019896000623703 /loss/FRt1/detection_loss‚1.3468984829092567 3loss/FBt2/assignment_loss‚0.0009845379390753806 0loss/FBt4/assignment_loss‚0.1520908554084599ß*_ã(Ù 3loss/FBt2/assignment_loss‚0.0009845379390753806 0loss/FBt4/assignment_loss‚0.1520908554084599 -loss/FBt2/detection_loss‚3.38371342420578 /loss/FBt3/detection_loss‚2.8464234200390903 % _timestamp‚1.7884369196408134e+09 0loss/FRt1/assignment_loss‚0.9433319243517789 epoch‚9 0loss/FRt4/assignment_loss‚3.0616043051083883 1loss/FBt1/assignment_loss‚0.18257178190964876 0loss/SRqqt1/detection_loss‚1.008178663691358 /loss/SRqqt4/detection_loss‚1.78962832562766 2loss/SRqqt4/assignment_loss‚0.9708299974225602 .loss/FRt4/detection_loss‚2.235518068406317 _runtime‚8604.345456913 0loss/FRt2/assignment_loss‚2.3300637722015383 2loss/SRqqt1/assignment_loss‚0.9510899087606323 1loss/SRqqt3/assignment_loss‚1.549165670518522 1loss/FBt3/assignment_loss‚0.33775490999956176 0loss/SRqqt2/detection_loss‚1.838340866799448 /loss/FBt1/detection_loss‚0.5785418762492089 _step‚934 trainer/global_step‚11349 2loss/SRqqt2/assignment_loss‚1.6607755093890078 /loss/FRt2/detection_loss‚2.2510289831293955 /loss/FRt3/detection_loss‚2.8560515353756566 %loss/total_loss‚40.53368811937854 /loss/FRt3/assignment_loss‚2.738526607713392 0loss/SRqqt3/detection_loss‚2.500678691599104 .loss/FBt4/detection_loss‚3.019896000623703 /loss/FRt1/detection_loss‚1.3468984829092567¦ôáÆÍ“ä(‚(:ˆ ½ÃåÔü”õŽ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.691' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9382 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K°¦u6å(‚(:« ½ÃåԠ÷# disk.dm-0.out‚51641.81494140625 disk.nvme0n1.in‚ 1076.453125( disk.nvme0c0n1.out‚51641.81494140625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 2.7541432e+07 network.recv‚ 2.936158e+06 disk.sda1.in‚0 disk.dm-0.in‚ 1076.453125" proc.memory.rssMB‚ 2872.3515625 disk.dm-1.in‚0.5 cpu‚25.35779666632448 disk.sda2.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚51641.81494140625 disk.rbd1.in‚0 disk.rbd1.out‚ 4.4736328125$ disk./.usageGB‚1100.824851989746 disk.sda.in‚0.5 disk.sda.out‚ 348.16015625) disk./.usagePercent‚73.87727947060428! disk.nvme0c0n1.in‚ 1076.453125! disk.rbd2.out‚227.03662109375 disk.drbd1777.out‚0* proc.memory.percent‚17.531442642211914 disk.dm-1.out‚ 348.16015625 disk.rbd0.out‚0 disk.drbd1777.in‚0$ memory_percent‚94.19286251068115( proc.memory.availableMB‚ 951.44140625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0.5 disk.sda3.out‚ 348.160156253ûÅœžæ(‚(:“ ÌÃåÔÓä± gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚155.691' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.138199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËUe´*ç(‚(:Ÿ ÌÃåÔ‚ô¹‘ disk.sda1.in‚0% disk.nvme0c0n1.out‚51741.65234375 disk.sda2.in‚0 disk.sda.out‚ 348.7578125& proc.memory.availableMB‚ 951.390625 cpu‚25.357285720878604# disk.nvme0n1.out‚51741.65234375! disk.nvme0c0n1.in‚ 1076.453125 disk.sda3.out‚ 348.7578125 disk.rbd3.out‚0 disk.dm-0.in‚ 1076.453125 disk.sda1.out‚0 disk.sda3.in‚0.5 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.out‚51741.65234375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0! disk.rbd2.out‚227.03662109375* proc.memory.percent‚17.531442642211914 network.recv‚ 2.942325e+06 disk.rbd2.in‚0" proc.memory.rssMB‚ 2872.3515625 proc.cpu.threads‚17 disk.rbd1.out‚ 4.4736328125 disk.sda2.out‚0) disk./.usagePercent‚73.87740798638646 disk.rbd3.in‚0 disk.dm-1.in‚0.5 disk.nvme0n1.in‚ 1076.453125 network.sent‚ 2.7649347e+07% disk./.usageGB‚1100.8267669677734 disk.sda.in‚0.5 disk.dm-1.out‚ 348.7578125$ memory_percent‚94.19317245483398ÆÙP«“è(‚(:ˆ ÛÃåÔËèý gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÙe„.é(‚(:£ ÛÃåÔŽ³· disk.rbd2.in‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚51841.93798828125 disk.sda.out‚349.3125 network.recv‚ 2.947111e+06 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1076.45703125 disk.rbd2.out‚ 227.138671875# memory_percent‚94.1962718963623 disk.dm-0.in‚ 1076.45703125 disk.dm-1.out‚349.3125& disk.nvme0n1.out‚51841.93798828125 network.sent‚ 2.7655743e+07 disk.sda.in‚0.5# disk.nvme0c0n1.in‚ 1076.45703125' proc.memory.availableMB‚ 950.8828125" proc.memory.rssMB‚ 2872.3515625 disk.sda3.in‚0.5 disk.dm-1.in‚0.5% disk./.usageGB‚1100.8265266418457 disk.sda3.out‚349.3125 disk.rbd3.in‚0 disk.sda1.in‚0* proc.memory.percent‚17.531442642211914( disk./.usagePercent‚73.8773918579118 disk.drbd1777.out‚0# disk.dm-0.out‚51841.93798828125 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0 cpu‚25.356788845379814 disk.rbd1.out‚ 4.53662109375 disk.sda2.in‚0ýFë2ê(- _wandbruntime‚8640  _runtime‚8640”£þÊë(Ä ' lr-AdamW/pg1‚2.2476093881369297e-06 ' lr-AdamW/pg2‚2.2476093881369297e-06 trainer/global_step‚11399 % _timestamp‚1.7884369556747928e+09 _runtime‚8640.379527202 _step‚935U–ë>Ïì(É trainer/global_step‚11399 % _timestamp‚1.7884369556747928e+09 _runtime‚8640.379527202 _step‚935 ' lr-AdamW/pg1‚2.2476093881369297e-06 ' lr-AdamW/pg2‚2.2476093881369297e-06§û<,ñ2í(- _wandbruntime‚8640  _runtime‚8640“)Tmê'î(äO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10019604810240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚655450859196222 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884369556755788e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9552348904960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165916998 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚724314581637632 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552348904960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚734334210387968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215311975 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734334024921088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 381228975 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151330301 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚724314605577728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚645934333402112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96808668 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚663324163679744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚645934350441472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9517180231866 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚672876530587648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 381208935 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165916998 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚672875856222208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551692542464 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552348904960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚734334210387968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215311975 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734334024921088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 381228975 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151330301 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚724314605577728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚645934333402112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96808668 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚663324163679744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚645934350441472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9517180231866 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚672876530587648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 381208935 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165916998 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚672875856222208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551692542464 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 381261471 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚734396530061312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚672933703067648 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215330455 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9552511970304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚655507251239070 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚645989254572032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 381241405 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚645989237532672 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884369563689723e+09 _runtime‚8641.074255198 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚663380516718592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165931014 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚724376256315392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151343371 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚11399 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165931016 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9553168346112 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151342933 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54526533 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚672933028688896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9552511970304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚937 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 381241405 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚663380534721536 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215310391 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165931016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚655506579761746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 381261471 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96816838 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9517342229074 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215330455 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚734396717612032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚724376232375296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10020297686016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9553168346112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944J˜lRù&ò(óM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚655507251239070 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚645989254572032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 381241405 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚645989237532672 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884369563689723e+09 _runtime‚8641.074255198 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚663380516718592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165931014 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚724376256315392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151343371 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚11399 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165931016 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9553168346112 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151342933 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54526533 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚672933028688896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9552511970304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚937 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 381241405 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚663380534721536 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215310391 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165931016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚655506579761746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 381261471 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96816838 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9517342229074 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215330455 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚734396717612032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚724376232375296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10020297686016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9553168346112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 381261471 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚734396530061312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚672933703067648 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215330455 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9552511970304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424© h&O2ó(- _wandbruntime‚8641  _runtime‚8641³æÀ[ô(Õ 1loss/SRqqt3/detection_loss‚2.2084733217190475 /loss/FBt4/detection_loss‚2.6286271756345574 trainer/global_step‚11399 _step‚938 /loss/FBt1/detection_loss‚0.5315682480321522 /loss/FRt1/detection_loss‚1.3668660798908345 /loss/FRt4/assignment_loss‚3.290051161001126 0loss/FBt3/assignment_loss‚0.1532885721535422 1loss/SRqqt2/assignment_loss‚1.664750188589096 2loss/SRqqt3/assignment_loss‚1.7904392070113084 1loss/SRqqt2/detection_loss‚1.9887806945376927 /loss/FBt3/detection_loss‚2.9180933237075806 epoch‚9 /loss/FRt2/detection_loss‚2.4233150634823777 0loss/FRt1/assignment_loss‚1.0460779911705427 0loss/FBt4/assignment_loss‚0.2709332565480674 _runtime‚8641.075452715 2loss/SRqqt1/assignment_loss‚0.7455761549597393 % _timestamp‚1.7884369563707845e+09 0loss/FRt2/assignment_loss‚2.8562405763602836 1loss/FBt1/assignment_loss‚0.13760826992034256 /loss/FBt2/assignment_loss‚0.173856729439509 .loss/FRt4/detection_loss‚2.172057669609785 0loss/SRqqt4/detection_loss‚1.741045865875024 2loss/SRqqt4/assignment_loss‚1.2451729937098348 1loss/SRqqt1/detection_loss‚0.8741891774364182 /loss/FRt3/assignment_loss‚2.969882086471275 /loss/FRt3/detection_loss‚3.0524177286359997 /loss/FBt2/detection_loss‚2.1833179730635424 %loss/total_loss‚40.43262950895968œV(t`õ(Ú /loss/FBt1/detection_loss‚0.5315682480321522 /loss/FRt1/detection_loss‚1.3668660798908345 /loss/FRt4/assignment_loss‚3.290051161001126 0loss/FBt3/assignment_loss‚0.1532885721535422 1loss/SRqqt2/assignment_loss‚1.664750188589096 2loss/SRqqt3/assignment_loss‚1.7904392070113084 1loss/SRqqt2/detection_loss‚1.9887806945376927 /loss/FBt3/detection_loss‚2.9180933237075806 epoch‚9 /loss/FRt2/detection_loss‚2.4233150634823777 0loss/FRt1/assignment_loss‚1.0460779911705427 0loss/FBt4/assignment_loss‚0.2709332565480674 _runtime‚8641.075452715 2loss/SRqqt1/assignment_loss‚0.7455761549597393 % _timestamp‚1.7884369563707845e+09 0loss/FRt2/assignment_loss‚2.8562405763602836 1loss/FBt1/assignment_loss‚0.13760826992034256 /loss/FBt2/assignment_loss‚0.173856729439509 .loss/FRt4/detection_loss‚2.172057669609785 0loss/SRqqt4/detection_loss‚1.741045865875024 2loss/SRqqt4/assignment_loss‚1.2451729937098348 1loss/SRqqt1/detection_loss‚0.8741891774364182 /loss/FRt3/assignment_loss‚2.969882086471275 /loss/FRt3/detection_loss‚3.0524177286359997 /loss/FBt2/detection_loss‚2.1833179730635424 %loss/total_loss‚40.43262950895968 1loss/SRqqt3/detection_loss‚2.2084733217190475 /loss/FBt4/detection_loss‚2.6286271756345574 trainer/global_step‚11399 _step‚938ª¹b^#ö(‚(:ˆ êÃåÔîÊŽŽ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚150.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0744 gpu.0.smdržÃpClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06¤Ï%÷(‚(:š êÃåÔ’ì­‘ disk.rbd3.out‚0 disk.sda.in‚0.5 disk.rbd0.out‚0$ memory_percent‚94.19498443603516* proc.memory.percent‚17.531442642211914# disk.nvme0c0n1.in‚ 1076.45703125! disk.nvme0c0n1.out‚ 51843.0625 disk.rbd0.in‚0) disk./.usagePercent‚73.87740389026591 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 350.15234375% proc.memory.availableMB‚ 951.09375 proc.cpu.threads‚17 disk.rbd2.out‚ 227.138671875 disk.dm-0.in‚ 1076.45703125 disk.sda.out‚ 350.15234375 cpu‚25.356253169295446 disk.sda1.out‚0 disk.nvme0n1.out‚ 51843.0625 disk.sda2.out‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2872.3515625 disk.sda3.out‚ 350.15234375 disk.dm-0.out‚ 51843.0625! disk.nvme0n1.in‚ 1076.45703125 network.recv‚ 2.950662e+06% disk./.usageGB‚1100.8267059326172 disk.sda3.in‚0.5 disk.drbd1777.out‚0 disk.dm-1.in‚0.5 disk.rbd1.in‚0 disk.rbd1.out‚ 4.53662109375 disk.sda2.in‚0 network.sent‚ 2.7660881e+07 disk.rbd3.in‚0·’/’ø(‚(:‡ ùÃåÔ٪Πgpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.615' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.323 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^æâ.ù(‚(:£ ùÃåÔû€ø disk.drbd1777.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0% disk./.usageGB‚1100.8253936767578 disk.dm-0.out‚51942.16796875 disk.sda.in‚0.5! disk.nvme0n1.in‚ 1076.45703125 disk.rbd1.in‚0 disk.sda3.in‚0.5 network.sent‚ 2.7769641e+07 disk.drbd1777.out‚0 disk.dm-0.in‚ 1076.45703125 disk.sda1.in‚0# disk.nvme0n1.out‚51942.16796875 disk.rbd1.out‚ 4.599609375 disk.rbd2.out‚ 228.791015625 disk.sda3.out‚ 350.6953125 disk.sda2.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.531442642211914 cpu‚25.355727485842987 network.recv‚ 2.957515e+06% disk.nvme0c0n1.out‚51942.16796875 disk.sda2.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1076.45703125) disk./.usagePercent‚73.87731582367414 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 350.6953125 disk.dm-1.in‚0.5$ memory_percent‚94.19522285461426 disk.dm-1.out‚ 350.6953125' proc.memory.availableMB‚ 951.0546875" proc.memory.rssMB‚ 2872.3515625mÃê\2ú(- _wandbruntime‚8677  _runtime‚8677ý6u¼Èû( & lr-AdamW/pg1‚1.935003938706134e-06 & lr-AdamW/pg2‚1.935003938706134e-06 trainer/global_step‚11449 % _timestamp‚1.7884369927889693e+09 _runtime‚8677.493837728 _step‚939[V×eÍü(Ç & lr-AdamW/pg1‚1.935003938706134e-06 & lr-AdamW/pg2‚1.935003938706134e-06 trainer/global_step‚11449 % _timestamp‚1.7884369927889693e+09 _runtime‚8677.493837728 _step‚939«–€F2ý(- _wandbruntime‚8677  _runtime‚8677*•²é'þ(ãO _step‚940 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166617800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216217989 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9593334710784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚648679539930112 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 382835787 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9592678348288 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151964894 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚737459481580032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 $ _timestamp‚1.788436992789834e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚675734493970432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚675734493970432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚727397142460928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10062363059200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 382855827 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 382855827 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9592678348288 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151964894 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚737459481580032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 $ _timestamp‚1.788436992789834e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚675734493970432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚675734493970432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0¬‰n€”2€)- _wandbruntime‚8678  _runtime‚8678C–gÐU#)îM OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166631814 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚666198168660992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10063383944704 trainer/global_step‚11449 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 382868257 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216236443 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚675791666437120 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166631816 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216236443 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚675792340815872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚675792340815872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216256507 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382888323 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚727458769258496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97225236 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9594154151936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚666198168660992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚737521983301120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚658292621629154 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166631814 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166631816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9593497776128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚648734461100032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚8678.194435612 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216256507 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97225238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚648734444060672 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151977448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9594154151936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚737522177143296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9558832006446 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 382888323 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10063214042624 % _timestamp‚1.7884369934888759e+09 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54752212 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚727458793198592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚941 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDevice¸œVŸStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382888323 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚727458769258496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97225236 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9594154151936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚666198168660992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚737521983301120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚658292621629154 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166631814 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166631816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9593497776128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚648734461100032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚8678.194435612 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216256507 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97225238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚648734444060672 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151977448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9594154151936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚737522177143296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9558832006446 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 382888323 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10063214042624 % _timestamp‚1.7884369934888759e+09 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54752212 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚727458793198592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚941 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚666198168660992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10063383944704 trainer/global_step‚11449 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 382868257­ýÞ2ƒ)- _wandbruntime‚8678  _runtime‚8678‚T ^„)Ø 0loss/FRt3/assignment_loss‚3.3563570056642806 1loss/SRqqt1/assignment_loss‚0.703757956383809 /loss/FBt1/detection_loss‚0.4665373497379257 /loss/FBt3/detection_loss‚2.4764927517284048 1loss/FBt2/assignment_loss‚0.12504001193732014 _runtime‚8678.195516459 _step‚942 2loss/SRqqt2/assignment_loss‚1.3789087900032218 1loss/FBt1/assignment_loss‚0.12795848790714706 /loss/FRt1/detection_loss‚1.5061024253002622 .loss/FRt2/detection_loss‚2.259978872278462 1loss/SRqqt4/detection_loss‚1.6959191817863315 .loss/FBt2/detection_loss‚2.447490985194842 epoch‚9 /loss/FRt2/assignment_loss‚2.698220864586208 2loss/SRqqt3/assignment_loss‚1.6981881701008037 2loss/FBt4/assignment_loss‚0.026336240648691143 /loss/FRt3/detection_loss‚2.7087461956909724 %loss/total_loss‚39.52386980700767 /loss/FBt4/detection_loss‚2.6912194405283247 trainer/global_step‚11449 0loss/FRt4/assignment_loss‚3.1081265543782433 .loss/FRt4/detection_loss‚2.111094285582387 0loss/SRqqt1/detection_loss‚0.824296189087724 1loss/SRqqt2/detection_loss‚1.7639318438103566 0loss/FRt1/assignment_loss‚1.0169900222258135 2loss/SRqqt4/assignment_loss‚1.4432320946864052 1loss/FBt3/assignment_loss‚0.21588012852323468 0loss/SRqqt3/detection_loss‚2.673063959236498 % _timestamp‚1.7884369934907799e+09Œoœc…)Ý 0loss/FRt4/assignment_loss‚3.1081265543782433 .loss/FRt4/detection_loss‚2.111094285582387 0loss/SRqqt1/detection_loss‚0.824296189087724 1loss/SRqqt2/detection_loss‚1.7639318438103566 0loss/FRt1/assignment_loss‚1.0169900222258135 2loss/SRqqt4/assignment_loss‚1.4432320946864052 1loss/FBt3/assignment_loss‚0.21588012852323468 0loss/SRqqt3/detection_loss‚2.673063959236498 % _timestamp‚1.7884369934907799e+09 0loss/FRt3/assignment_loss‚3.3563570056642806 1loss/SRqqt1/assignment_loss‚0.703757956383809 /loss/FBt1/detection_loss‚0.4665373497379257 /loss/FBt3/detection_loss‚2.4764927517284048 1loss/FBt2/assignment_loss‚0.12504001193732014 _runtime‚8678.195516459 _step‚942 2loss/SRqqt2/assignment_loss‚1.3789087900032218 1loss/FBt1/assignment_loss‚0.12795848790714706 /loss/FRt1/detection_loss‚1.5061024253002622 .loss/FRt2/detection_loss‚2.259978872278462 1loss/SRqqt4/detection_loss‚1.6959191817863315 .loss/FBt2/detection_loss‚2.447490985194842 epoch‚9 /loss/FRt2/assignment_loss‚2.698220864586208 2loss/SRqqt3/assignment_loss‚1.6981881701008037 2loss/FBt4/assignment_loss‚0.026336240648691143 /loss/FRt3/detection_loss‚2.7087461956909724 %loss/total_loss‚39.52386980700767 /loss/FBt4/detection_loss‚2.6912194405283247 trainer/global_step‚11449® ¾ÿ\†)‚(:’ ˆÄåÔÏÆ€Ž gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.74' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.148000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€Ué%‡)‚(:š ˆÄåÔ·Ë‘ disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1076.45703125 disk.drbd1777.in‚0" proc.memory.rssMB‚ 2872.3515625 cpu‚25.355225754698004 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd1777.out‚0! disk.nvme0n1.in‚ 1076.45703125 disk.rbd1.out‚ 4.599609375 disk.sda.in‚0.5% disk.nvme0n1.out‚52041.4638671875 disk.sda3.in‚0.5 disk.rbd3.in‚0' proc.memory.availableMB‚ 951.0234375 proc.cpu.threads‚17) disk./.usagePercent‚73.87731582367414 disk.dm-1.out‚351.25' disk.nvme0c0n1.out‚52041.4638671875 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 228.791015625 disk.rbd3.out‚0% disk./.usageGB‚1100.8253936767578" disk.dm-0.out‚52041.4638671875 disk.rbd0.out‚0$ memory_percent‚94.19541358947754 network.recv‚ 2.961554e+06 disk.dm-0.in‚ 1076.45703125 disk.sda.out‚351.25 disk.dm-1.in‚0.5 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda3.out‚351.25* proc.memory.percent‚17.531442642211914 network.sent‚ 2.7775146e+079·n<“ˆ)‚(:ˆ —ÄåÔ¯¢ŒŽ gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.591' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.3182 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀøÔb4‰)‚(:© —ÄåÔø’Ì& disk.nvme0n1.out‚52142.43408203125 disk.sda.in‚0.5 disk.dm-1.in‚0.5 disk.rbd2.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.531442642211914) disk./.usagePercent‚73.87742411486113 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 228.931640625 disk.drbd1777.out‚0# disk.dm-0.out‚52142.43408203125 disk.sda1.in‚0 disk.sda3.out‚ 351.8359375# memory_percent‚94.1976547241211 disk.sda.out‚ 351.8359375 disk.drbd1777.in‚0% proc.memory.availableMB‚ 950.65625% disk./.usageGB‚1100.8270072937012 disk.dm-0.in‚ 1076.45703125( disk.nvme0c0n1.out‚52142.43408203125 disk.sda3.in‚0.5 proc.cpu.threads‚17 network.recv‚ 2.968284e+06! disk.nvme0n1.in‚ 1076.45703125 disk.rbd1.out‚ 4.599609375# disk.nvme0c0n1.in‚ 1076.45703125 disk.sda2.in‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2872.3515625 cpu‚25.354678365848805 disk.sda2.out‚0 disk.rbd0.in‚0 network.sent‚ 2.7884039e+07 disk.sda1.out‚0 disk.dm-1.out‚ 351.8359375‡†«œ2Š)- _wandbruntime‚8714  _runtime‚8714³XIÉ‹)à ' lr-AdamW/pg2‚1.6456091639793144e-06 trainer/global_step‚11499 $ _timestamp‚1.788437030155021e+09 _runtime‚8714.859764253 _step‚943 ' lr-AdamW/pg1‚1.6456091639793144e-06n[‘<ÎŒ)È $ _timestamp‚1.788437030155021e+09 _runtime‚8714.859764253 _step‚943 ' lr-AdamW/pg1‚1.6456091639793144e-06 ' lr-AdamW/pg2‚1.6456091639793144e-06 trainer/global_step‚11499¯|GÁ†2)- _wandbruntime‚8714  _runtime‚8714N»9ê'Ž)äO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚668959467564544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678593131718656 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚730479679344128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661022942931038 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚668959485567488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9634320516608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚740585112144384 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167318600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9634320516608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54974921 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚651424746458112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚11499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 384462639 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚678593806084096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167318598 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚740585305999872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚730479655404032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9633664154112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884370301557999e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚651424763497472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚668959485567488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97625468 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9598196472926 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217164079 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217144041 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚661023614408154 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 384482679 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9598850910682 _runtime‚8714.860891316 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚678593806084096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10105626655744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217144041 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚678593131718656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167318598 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167318600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97625470 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152600389 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚668959467564544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756–מ)éO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚678593131718656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167318598 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167318600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97625470 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152600389 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚668959467564544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚668959467564544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678593131718656 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚730479679344128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661022942931038 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚668959485567488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9634320516608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚740585112144384 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167318600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9634320516608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54974921 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚651424746458112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚11499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 384462639 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚678593806084096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167318598 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚740585305999872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚730479655404032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9633664154112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884370301557999e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚651424763497472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚668959485567488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97625468 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9598196472926 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217164079 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217144041 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚661023614408154 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 384482679 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9598850910682 _runtime‚8714.860891316 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚678593806084096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10105626655744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217144041 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889°j…rØž)‚(:“ ¦ÄåÔÏøæ gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚151.553' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.310599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îðÁ—%‘)‚(:š ¦ÄåÔ¸›Ç disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 352.4609375 disk.rbd1.in‚0% disk./.usageGB‚1100.8286323547363 disk.rbd0.out‚0 disk.drbd1777.in‚0$ memory_percent‚94.19918060302734" proc.memory.rssMB‚ 2872.3515625( disk./.usagePercent‚73.8775331740707# disk.nvme0c0n1.in‚ 1076.45703125 disk.sda3.in‚0.5 disk.rbd3.in‚0 disk.dm-0.in‚ 1076.45703125 disk.dm-0.out‚ 52143.453125! disk.nvme0n1.out‚ 52143.453125% proc.memory.availableMB‚ 950.40625 disk.sda3.out‚ 352.4609375# disk.nvme0c0n1.out‚ 52143.453125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 228.931640625 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1076.45703125 disk.rbd1.out‚ 4.599609375 disk.sda2.out‚0 disk.sda.in‚0.5 disk.sda1.out‚0 disk.dm-1.in‚0.5 disk.dm-1.out‚ 352.4609375 proc.cpu.threads‚17 network.recv‚ 2.972085e+06 disk.sda1.in‚0 network.sent‚ 2.7889237e+07* proc.memory.percent‚17.531442642211914 cpu‚25.3541843697723624<“·2’)- _wandbruntime‚8715  _runtime‚8715¤îk4ô&“)îM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9635139957760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97633636 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚669015820603392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚678650304185344 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚651479650588672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97633638 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152613121 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚661078663496562 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚945 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167332616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10106308040192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚669015838606336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚740647614181888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚678650978564096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152613577 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9599667345854 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚730541330081792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384515175 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884370308444638e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚678650978564096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚730541306141696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9599012907890 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167332614 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217182559 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11499 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167332614 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9634483581952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872×Xsù&”)óM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚678650978564096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152613577 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9599667345854 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚730541330081792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384515175 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884370308444638e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚678650978564096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚730541306141696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9599012907890 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167332614 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217182559 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11499 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167332614 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9634483581952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9635139957760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97633636 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚669015820603392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚678650304185344 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚651479650588672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97633638 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152613121 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚661078663496562 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚945 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167332616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10106308040192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚669015838606336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚740647614181888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844±Ïýù­2•)- _wandbruntime‚8715  _runtime‚8715Xà£WY–)Ó 1loss/SRqqt1/detection_loss‚0.8911494583464585 0loss/FBt1/detection_loss‚0.49493377599405286 $ _timestamp‚1.788437030846589e+09 0loss/FBt4/assignment_loss‚0.0815033618774679 .loss/FRt2/detection_loss‚2.458559255066671 1loss/FBt2/assignment_loss‚0.07598485942435218 2loss/FBt3/assignment_loss‚0.002853766859819492 1loss/SRqqt2/detection_loss‚1.7007348829949345 .loss/FBt4/detection_loss‚2.280387202898661 %loss/total_loss‚39.72357455485724 epoch‚9 /loss/FRt3/assignment_loss‚3.087789113180978 0loss/SRqqt2/assignment_loss‚1.26271242188307 0loss/SRqqt3/detection_loss‚2.197852532068888 _runtime‚8715.551260015 1loss/FBt1/assignment_loss‚0.15866745658801604 .loss/FBt3/detection_loss‚3.293741762638092 0loss/FRt1/assignment_loss‚1.4059134550166852 /loss/FRt4/assignment_loss‚3.053088217973709 0loss/FRt2/assignment_loss‚1.9487992274133783 2loss/SRqqt3/assignment_loss‚1.6084186877124012 /loss/FRt3/detection_loss‚2.6708186541284835 .loss/FBt2/detection_loss‚3.291193649172783 trainer/global_step‚11499 _step‚946 1loss/SRqqt1/assignment_loss‚0.877430466025208 2loss/SRqqt4/assignment_loss‚1.3773042970690235 /loss/FRt4/detection_loss‚2.3524624783059824 0loss/SRqqt4/detection_loss‚1.593631813238407 /loss/FRt1/detection_loss‚1.5576437589797107ûêo^—)Ø /loss/FRt3/assignment_loss‚3.087789113180978 0loss/SRqqt2/assignment_loss‚1.26271242188307 0loss/SRqqt3/detection_loss‚2.197852532068888 _runtime‚8715.551260015 1loss/FBt1/assignment_loss‚0.15866745658801604 .loss/FBt3/detection_loss‚3.293741762638092 0loss/FRt1/assignment_loss‚1.4059134550166852 /loss/FRt4/assignment_loss‚3.053088217973709 0loss/FRt2/assignment_loss‚1.9487992274133783 2loss/SRqqt3/assignment_loss‚1.6084186877124012 /loss/FRt3/detection_loss‚2.6708186541284835 .loss/FBt2/detection_loss‚3.291193649172783 trainer/global_step‚11499 _step‚946 1loss/SRqqt1/assignment_loss‚0.877430466025208 2loss/SRqqt4/assignment_loss‚1.3773042970690235 /loss/FRt4/detection_loss‚2.3524624783059824 0loss/SRqqt4/detection_loss‚1.593631813238407 /loss/FRt1/detection_loss‚1.5576437589797107 1loss/SRqqt1/detection_loss‚0.8911494583464585 0loss/FBt1/detection_loss‚0.49493377599405286 $ _timestamp‚1.788437030846589e+09 0loss/FBt4/assignment_loss‚0.0815033618774679 .loss/FRt2/detection_loss‚2.458559255066671 1loss/FBt2/assignment_loss‚0.07598485942435218 2loss/FBt3/assignment_loss‚0.002853766859819492 1loss/SRqqt2/detection_loss‚1.7007348829949345 .loss/FBt4/detection_loss‚2.280387202898661 %loss/total_loss‚39.72357455485724 epoch‚9² T—i“˜)‚(:ˆ µÄåÔìú’Ž gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.188' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.6376 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Ðú,™)‚(:¡ µÄåÔéÄŒ‘ network.sent‚ 2.7986722e+07% disk./.usageGB‚1100.8285751342773 disk.rbd2.in‚0 disk.sda.in‚0.5 disk.sda1.in‚0 disk.dm-1.out‚ 353.265625( disk./.usagePercent‚73.8775293339577 disk.sda3.in‚0.5 disk.dm-1.in‚0.5 network.recv‚ 2.979209e+06# disk.nvme0n1.out‚52242.55859375 disk.rbd0.out‚0 disk.sda3.out‚ 353.265625 disk.sda1.out‚0! disk.nvme0n1.in‚ 1076.45703125* proc.memory.percent‚17.531442642211914% disk.nvme0c0n1.out‚52242.55859375 disk.sda2.in‚0! disk.rbd2.out‚229.03369140625 disk.rbd1.in‚0$ memory_percent‚94.19798851013184 cpu‚25.353660023408477 disk.sda2.out‚0 disk.sda.out‚ 353.265625 disk.rbd1.out‚ 4.599609375' proc.memory.availableMB‚ 950.6015625 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1076.45703125 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.dm-0.out‚52242.55859375" proc.memory.rssMB‚ 2872.3515625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 1076.45703125Y~ò¤žš)‚(:“ ÄÄåÔ°¿ê gpu.0.gpu‚10.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚149.304' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.860799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™û™9›)‚(:® ÄÄåÔêÇÀ( disk.nvme0c0n1.out‚52342.02294921875 disk.rbd2.in‚0 network.sent‚ 2.8084005e+07% disk./.usageGB‚1100.8285751342773 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2872.3515625 disk.dm-1.out‚ 353.7421875 disk.rbd1.in‚0 disk.rbd1.out‚ 4.599609375# disk.nvme0c0n1.in‚ 1076.45703125 disk.sda3.out‚ 353.7421875# disk.dm-0.out‚52342.02294921875& disk.nvme0n1.out‚52342.02294921875 disk.sda2.out‚0 disk.rbd0.out‚0( disk./.usagePercent‚73.8775293339577 disk.dm-0.in‚ 1076.45703125( proc.memory.availableMB‚ 950.58984375 cpu‚25.353144248327318 proc.cpu.threads‚17 network.recv‚ 2.985258e+06 disk.sda3.in‚0.5 disk.rbd3.in‚0 disk.sda.in‚0.5 disk.dm-1.in‚0.5$ memory_percent‚94.19806003570557* proc.memory.percent‚17.531442642211914 disk.drbd1777.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1076.45703125 disk.rbd0.in‚0! disk.rbd2.out‚229.03369140625 disk.sda.out‚ 353.7421875 disk.sda1.in‚0=QS¼2œ)- _wandbruntime‚8751  _runtime‚8751×·3óÊ)Ä ' lr-AdamW/pg1‚1.3794853733937692e-06 ' lr-AdamW/pg2‚1.3794853733937692e-06 trainer/global_step‚11549 % _timestamp‚1.7884370672304492e+09 _runtime‚8751.935232778 _step‚947èêtÏž)É ' lr-AdamW/pg1‚1.3794853733937692e-06 ' lr-AdamW/pg2‚1.3794853733937692e-06 trainer/global_step‚11549 % _timestamp‚1.7884370672304492e+09 _runtime‚8751.935232778 _step‚947³#„‹‚ Ÿ)- _waœìŠ›&ndbruntime‚8751  _runtime‚8751(`¯ê' )äO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚743710816734720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218090131 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218070093 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98033870 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168019400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚733562192287232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚8751.936307402 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚654169970025472 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168019400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10148600507392 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218070093 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚733562216227328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚654169952986112 _step‚948 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚671777137509888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55201285 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚743710622879232 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168019398 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218090131 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚663808984798446 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚663809656275562 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168019398 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10148430592000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚681452443832320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚681451769466880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 386109531 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55200827 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9639031812334 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚671777119506944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚681452443832320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9675306322432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9639686250090 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9675306322432 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153234695 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153235155 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681451769466880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚671777119506944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9674649959936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872f¸%ï'¡)éO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚733562192287232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚8751.936307402 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚654169970025472 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168019400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10148600507392 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218070093 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚733562216227328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚654169952986112 _step‚948 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚671777137509888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55201285 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚743710622879232 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168019398 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218090131 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚663808984798446 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚663809656275562 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168019398 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10148430592000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚681452443832320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚681451769466880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 386109531 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55200827 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9639031812334 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚671777119506944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚681452443832320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9675306322432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9639686250090 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9675306322432 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153234695 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153235155 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681451769466880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚671777119506944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9674649959936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚743710816734720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218090131 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218070093 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98033870 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168019400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10149285305856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9639848247298 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168033414 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55205425 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚733623843024896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚671833472545792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚949 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10149451013632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 386142027 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚654224874156032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218088547 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚743773317978624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 trainer/global_step‚11549 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚8752.631910428 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚671833490548736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9675469387776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168033416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚654224857116672 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 386142027 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9676125763584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚743773128330752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚663865376841294 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚671833472545792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9640502685262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚671833490548736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 386121961 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153247461 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚733623866964992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153247905 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9676125763584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218108611-Û!³¤)ñM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153247905 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9676125763584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218108611 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚1014928‡9¾D5305856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9639848247298 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168033414 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55205425 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚733623843024896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚671833472545792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚949 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10149451013632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 386142027 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚654224874156032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218088547 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚743773317978624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 trainer/global_step‚11549 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚8752.631910428 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚671833490548736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9675469387776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168033416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚654224857116672 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 386142027 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9676125763584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚743773128330752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚663865376841294 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚671833472545792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9640502685262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚671833490548736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 386121961 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153247461 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚733623866964992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚684311081580544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚674594771449344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚11599 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168720198 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚684310407215104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218996145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98442270 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55426315 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚674594789452288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10191584698368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10191423171584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚666595026665854 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9716292128256 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚674594771449344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387736383 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚656915176553472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚736644753110528 _step‚952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656915159514112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219016183 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153869023 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387716343 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚746836152342016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚684311081580544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9715635765760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚684310407215104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚674594789452288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9679867151742 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038©1óýÝ ±)éO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9715635765760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9716292128256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚736644729170432 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚666595698142970 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168720198 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚684311081580544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚674594771449344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚11599 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168720198 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚684310407215104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218996145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98442270 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55426315 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚674594789452288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10191584698368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10191423171584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚666595026665854 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9716292128256 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚674594771449344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387736383 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚656915176553472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚736644753110528 _step‚952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656915159514112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219016183 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153869023 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387716343 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚746836152342016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚684311081580544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9715635765760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚684310407215104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚674594789452288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9679867151742 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153868583 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98442268 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55426753 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚746836337808896 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218996145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219016183 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884371044702237e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168720200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756¸îÂ÷ì2²)- _wandbruntime‚8789  _runtime‚8789ácIô&³)îM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387768879 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚746898660874752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚736706403848192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10192280966656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387768879 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387748813 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9717111569408 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153881214 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚746898852619776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168734214 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚674651124488192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚684368254060544 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153881665 % _timestamp‚1.7884371051612868e+09 _runtime‚8789.867001504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9717111569408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚674651142491136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656970063644672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚666650747231378 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9681338024670 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168734216 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98450436 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55431227 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55430778 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219014599 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9716455193600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚674651142491136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168734216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚736706379908096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414`Ú¦ ù&´)óM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚656970080684032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚674651142491136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656970063644672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚666650747231378 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9681338024670 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168734216 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98450436 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55431227 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55430778 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219014599 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9716455193600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚674651142491136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168734216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚736706379908096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387768879 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚746898660874752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚736706403848192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10192280966656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387768879 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387748813 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9717111569408 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153881214 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚746898852619776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168734214 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚674651124488192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚684368254060544 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153881665 % _timestamp‚1.7884371051612868e+09 _runtime‚8789.867001504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9717111569408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F`c·)Ý %loss/total_loss‚37.66758281579894 epoch‚9 0loss/FRt1/assignment_loss‚1.1830190214243803 2loss/SRqqt3/assignment_loss‚1.3195218719980295 1loss/FBt2/assignment_loss‚0.23931554835417046 1loss/FBt3/assignment_loss‚0.01850850916231366 1loss/SRqqt4/detection_loss‚1.8306943276580774 /loss/FBt4/detection_loss‚2.0838405815037815 2loss/SRqqt4/assignment_loss‚1.0140213252062786 % _timestamp‚1.7884371051634858e+09 2loss/FBt1/assignment_loss‚0.029744207878326948 0loss/FBt4/assignment_loss‚0.1141107241538438 /loss/FRt1/detection_loss‚1.7459018177606842 /loss/FRt2/detection_loss‚2.7243025633128912 0loss/SRqqt1/detection_loss‚1.041180985255374 /loss/FRt3/detection_loss‚2.9188257413251057 1loss/SRqqt2/detection_loss‚1.8665699001688223 trainer/global_step‚11599 _step‚954 _runtime‚8789.868167962 /loss/FRt3/assignment_loss‚2.837360360792705 2loss/SRqqt1/assignment_loss‚0.6795382263242371 1loss/SRqqt2/assignment_loss‚1.321785360467262 0loss/SRqqt3/detection_loss‚2.273726288010092 0loss/FBt1/detection_loss‚0.38522926830303145 /loss/FBt2/detection_loss‚2.7611089795827866 /loss/FRt4/assignment_loss‚2.424341230857663 /loss/FRt2/assignment_loss‚2.007312568458351 .loss/FRt4/detection_loss‚2.502286911010742 /loss/FBt3/detection_loss‚2.3453364968299866ºJ½…ž¸)‚(:“ ñÄåÔ¡ªß gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚135.908' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.181599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FÊd19¹)‚(:® ñÄåÔ¥úþ disk.sda3.in‚0.5 disk.rbd2.in‚0 disk.rbd2.out‚230.8310546875 disk.sda1.in‚0* proc.memory.percent‚17.531442642211914 cpu‚25.351622348718102 disk.rbd0.in‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 1076.45703125( proc.memory.availableMB‚ 950.66796875" proc.memory.rssMB‚ 2872.3515625) disk./.usagePercent‚73.87753061399536 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda.in‚0.5$ memory_percent‚94.19758319854736# disk.dm-0.out‚52540.94873046875 disk.sda.out‚ 355.7734375! disk.nvme0n1.in‚ 1076.45703125( disk.nvme0c0n1.out‚52540.94873046875 disk.sda3.out‚ 355.7734375 network.sent‚ 2.8204378e+07 network.recv‚ 3.000173e+06 disk.rbd3.in‚0 disk.dm-1.in‚0.5 disk.rbd1.in‚0 disk.dm-1.out‚ 355.7734375 disk.rbd1.out‚ 4.599609375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 1076.45703125% disk./.usageGB‚1100.8285942077637 disk.sda1.out‚0& disk.nvme0n1.out‚52540.94873046875 proc.cpu.threads‚17¢”¶Qžº)‚(:“ €ÅåÔª¹Ž gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.041' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.008199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶!~K:»)‚(:¯ €ÅåÔÀäÕ# disk.nvme0c0n1.in‚ 1076.45703125" proc.memory.rssMB‚ 2872.3515625 disk.dm-0.in‚ 1076.45703125 disk.dm-1.out‚ 356.64453125! disk.nvme0n1.in‚ 1076.45703125 disk.sda2.out‚0( disk./.usagePercent‚73.8775472544851# disk.dm-0.out‚52640.42041015625& disk.nvme0n1.out‚52640.42041015625 disk.rbd1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.531442642211914 disk.rbd2.out‚230.8310546875 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚52640.42041015625 disk.sda.out‚ 356.64453125 disk.dm-1.in‚0.5 disk.sda3.out‚ 356.64453125 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 950.41015625 network.sent‚ 2.8312525e+07 disk.sda3.in‚0.5$ memory_percent‚94.19915676116943 network.recv‚ 3.006343e+06$ disk./.usageGB‚1100.828842163086 disk.sda.in‚0.5 disk.rbd2.in‚0 cpu‚25.351111942860616 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 4.599609375ß@»s“¼)‚(:ˆ ÅåÔåØ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ï>è0½)‚(:¥ ÅåÔçÛç disk.dm-1.in‚0.5 disk.sda2.in‚0 disk.sda3.in‚0.5 disk.sda3.out‚ 357.2421875 disk.drbd1777.out‚0* proc.memory.percent‚17.531442642211914) disk./.usagePercent‚73.87754597444741 disk.nvme0n1.in‚ 1076.484375 disk.sda2.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1076.484375" disk.dm-0.out‚52739.5751953125 disk.sda.out‚ 357.2421875 network.sent‚ 2.8318867e+07 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 1076.484375 disk.rbd0.out‚0( proc.memory.availableMB‚ 950.50390625 cpu‚25.350606474796525 proc.cpu.threads‚17 disk.dm-1.out‚ 357.2421875% disk.nvme0n1.out‚52739.5751953125 disk.sda1.in‚0" proc.memory.rssMB‚ 2872.3515625 disk.rbd1.out‚ 4.599609375 disk.rbd0.in‚0 disk.sda.in‚0.5$ memory_percent‚94.19858455657959 disk.rbd2.out‚230.9248046875% disk./.usageGB‚1100.8288230895996' disk.nvme0c0n1.out‚52739.5751953125 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 network.recv‚ 3.011152e+06 disk.rbd2.in‚0C×|+2¾)- _wandbruntime‚8825  _runtime‚8825 ›ÇÉÈ¿) % _timestamp‚1.7884371411214683e+09 _runtime‚8825.826182517 _step‚955 & lr-AdamW/pg1‚9.172677226469214e-07 & lr-AdamW/pg2‚9.172677226469214e-07 trainer/global_step‚11649¨æTÄÍÀ)Ç & lr-AdamW/pg1‚9.172677226469214e-07 & lr-AdamW/pg2‚9.172677226469214e-07 trainer/global_step‚11649 % _timestamp‚1.7884371411214683e+09 _runtime‚8825.826182517 _step‚955»Wb,í2Á)- _wandbruntime‚8825  _runtime‚8825¶ØUŽó Â)ãO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389363235 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚687169719328768 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚687169044963328 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169421000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚669381068533262 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚687169719328768 _runtime‚8825.827116898 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚687169044963328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚677412441394688 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9757277934080 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚749961839418880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11649 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚677412423391744 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9756621571584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98850670 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚749961649757696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚739727289993728 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10234383704064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98850668 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55652635 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 $ _timestamp‚1.788437141122155e+09ÐRÙ¼î'Ã)èO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389363235 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚687169719328768 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚687169044963328 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169421000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚669381068533262 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚687169719328768 _runtime‚8825.827116898 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚687169044963328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚677412441394688 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9757277934080 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚749961839418880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11649 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚677412423391744 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9756621571584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98850670 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚749961649757696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚739727289993728 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10234383704064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98850668 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55652635 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 $ _timestamp‚1.788437141122155e+09 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9757440999424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389395731 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154515483 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55657107 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219940651 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884371417869134e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚687226217430016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219960715 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389395731 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚687226891808768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154515945 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚750024352781824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10235412050432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389375665 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚957 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚739788940731392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324£ù| ÓÆ)óM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98858836 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚677468776430592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9758097375232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚11649 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚957 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚739788940731392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚659715270172672 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169435016 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169435014 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚750024163133952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9757440999424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389395731 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154515483 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55657107 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219940651 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884371417869134e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚687226217430016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 EÝ&rain_batch_end/inactive_split_bytes.small_pool.freed‚10235246342656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚669436789098786 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219960715 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389395731 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚687226891808768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154515945 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚750024352781824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10235412050432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389375665 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690027682711552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170121800 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10277462359040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚662405572570112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚742809826876928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚742809802936832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220848249 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9761537830558 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155137502 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884371765519083e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690028357076992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55878434 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚680230093337088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690028357076992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚662405589609472 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚680230075334144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 390990087 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220868287 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9798263739904 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9762192268314 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚ 8861.25692172 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9798263739904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99259068 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚672167110400670(í"î'Ó)èO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10277462359040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚662405572570112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚742809826876928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚742809802936832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220848249 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9761537830558 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155137502 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884371765519083e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690028357076992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55878434 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚680230093337088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690028357076992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚662405589609472 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚680230075334144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 390990087 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220868287 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9798263739904 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9762192268314 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚ 8861.25692172 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9798263739904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99259068 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚672167110400670 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170121800 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155137928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 390970047 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690027682711552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170121800 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040ÀIÉ-•2Ô)- _wandbruntime‚8861  _runtime‚8861GÀÐÄúÕ)ìM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 FDeviceStatÎmªø!sMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚680286428372992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚742871453674496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10278336767488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155150144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99267238 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170135816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391022583 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155150572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚680286446375936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚753149790441984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9763008703486 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9799083181056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚753149971701248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170135816 _step‚961 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚680286428372992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220886767 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9762354265522 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 391022583 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9798426805248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚690085529556992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9798426805248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚690084855178240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚742871477614592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55883334 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 391002517 _runtime‚8861.936147544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 # _timestamp‚1.78843717723093e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚672223502443518 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170135814 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚672222830966194 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 391002517 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10278494086656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚11699 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚690085529556992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220886767 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220866703 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚680286428372992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚742871453674496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10278336767488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155150144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99267238 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170135816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391022583 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155150572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚680286446375936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚753149790441984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9763008703486 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9799083181056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚753149971701248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170135816 _step‚961 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚680286428372992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220886767 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9762354265522 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 391022583 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9798426805248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚690085529556992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9798426805248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚690084855178240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚742871477614592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55883334 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 391002517 _runtime‚8861.936147544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 # _timestamp‚1.78843717723093e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚672223502443518 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170135814 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚672222830966194 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 391002517 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10278494086656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚11699 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚690085529556992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220886767 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220866703 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚665150779098112 _step‚964 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚11749 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚692886994825216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99667468 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚692886320459776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56106302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9803027607722 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221794339 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9839249545728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚8896.788503826 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚683047745279488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170822598 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221794339 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚756212913227264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155773770 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚683047727276544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844Y|ñ‰ï'å)éO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚665150779098112 _step‚964 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚11749 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚692886994825216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99667468 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚692886320459776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56106302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9803027607722 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221794339 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9839249545728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚8896.788503826 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚683047745279488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170822598 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221794339 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚756212913227264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155773770 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚683047727276544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9838593183232 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 392616939 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚692886320459776 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221774301 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚683047745279488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9838593183232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10320392134656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ÄéX„2æ)- _wandbruntime‚8897  _runtime‚8897²öpô&ç)îM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9840068986880 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚692943492926464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚683104080315392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 392629369 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170836616 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155786852 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221792755 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155786415 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚692944167305216 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚11749 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392649435 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170836616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚965 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 392649435 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683104080315392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56110779 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221812819 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚692944167305216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9840068986880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚675009544310926 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884372127412336e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170836614 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170836614 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚756275429575168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚745954014497792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10321415077376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10321253563904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665205683228672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221792755 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚756275244121600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚745953990557696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221812819 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚665205700268032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚675008872833602 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9803189604930 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392649435 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170836616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚965 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 392649435 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683104080315392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56110779 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221812819 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚692944167305216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9840068986880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚675009544310926 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884372127412336e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170836614 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170836614 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚756275429575168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚745954014497792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10321415077376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10321253563904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665205683228672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221792755 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚756275244121600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚745953990557696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221812819 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚665205700268032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚675008872833602 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9803189604930 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚683104080315392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 392629369 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170836616 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155786852 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_poolÍÌBw·.peak‚ 868220928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221792755 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155786415 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚692944167305216 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚11749 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚695744958208000 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156410295 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100075868 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9843862947130 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222700353 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚695744958208000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222720391 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚748974876703232 % _timestamp‚1.7884372474274075e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 394243791 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚677739194135486 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 394223751 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9879578989056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚685865379218944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚759338453912576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9880235351552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9880235351552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚695745632573440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56334853 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚685865397221888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748974900643328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171523400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚685865397221888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚695744958208000 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156410295 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100075868 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9843862947130 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222700353 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚695744958208000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222720391 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚748974876703232 % _timestamp‚1.7884372474274075e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 394243791 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚677739194135486 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 394223751 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9879578989056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚685865379218944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚759338453912576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9880235351552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9880235351552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚695745632573440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56334853 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚685865397221888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748974900643328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171523400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚685865397221888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8932.801340328 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 171537416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 # _timestamp‚1.78843724809617e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚685921750260736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚695802130674688 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156423054 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9881054792704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56339462 _step‚969 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685921750260736 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100084036 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 REDeviceStatsMonitor.on_train_batch_end/inactive_split.smallæCõ£¼_pool.freed‚56339018 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171537414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685921732257792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚695802805053440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚677795586178334 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 394256221 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚11799 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100084038 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222738871 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171537414 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 394256221 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156423500 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 394276287 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚667950906796032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚667950889756672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚685921732257792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8932.801340328 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 171537416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 # _timestamp‚1.78843724809617e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚685921750260736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚695802130674688 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156423054 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9881054792704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56339462 _step‚969 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685921750260736 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100084036 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56339018 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171537414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685921732257792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚695802805053440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚677795586178334 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 394256221 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚11799 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100084038 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222738871 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171537414 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 394256221 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156423500 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 394276287 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚667950906796032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚667950889756672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚685921732257792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395870643 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9921221157376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157046021 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚670641192154112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚688683049164288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚762463953640960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10406712067072 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172224198 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100484268 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚680525236002894 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9884698286538 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚11849 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9921221157376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚762464149593600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223626405 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698603595956224 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157046479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚670641209193472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚688683031161344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10406540054528 _step‚972 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9920564794880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395850603 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56561753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚680525907480010 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚688683031161344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223646443 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚8967.535157837 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚680525907480010 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚688683031161344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223646443 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚8967.535157837 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395870643 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9921221157376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157046021 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚670641192154112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚688683049164288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚762463953640960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10406712067072 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172224198 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100484268 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚680525236002894 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9884698286538 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚11849 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9921221157376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚762464149593600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223626405 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698603595956224 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157046479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚670641209193472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚688683031161344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458̧Xg2†*- _wandbruntime‚8968  _runtime‚8968ëà ö&‡*ðM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56566244 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172238214 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157058680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚11849 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223644859 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172238216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172238216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚680581628045742 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395903139 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚762526474612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9885514721710 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 395883073 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9922040598528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223664923 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10407567607296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚752119088264192 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670696096284672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395903139 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10407410288128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9921384222720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚8968.195924884 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚762526655871488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395883073 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223644859 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100492436 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚688739402203136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9921384222720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172238214 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9884860283746 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56566683 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚670696113324032 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157059121 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚698660768422912 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚752119064324096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0“È…)û&ˆ*õM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100492438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223664923 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚698661442801664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚688739402203136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9922040598528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚680580956568418 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395903139 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10407410288128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9921384222720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚8968.195924884 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚762526655871488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395883073 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223644859 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100492436 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚688739402203136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9921384222720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172238214 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9884860283746 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56566683 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚670696113324032 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157059121 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚698660768422912 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚752119064324096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56566244 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172238214 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157058680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚11849 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223644859 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172238216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172238216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚680581628045742 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395903139 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚762526474612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9885514721710 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 395883073 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9922040598528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223664923 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10407567607296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚752119088264192 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670696096284672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10449571302912 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224572495 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172925000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚691500701106688 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172924998 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100892670 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9924879188190 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚701462908069888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚765589703045120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172925000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚9002.739906535 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚691500683103744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397497495 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚701462233704448 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224572495 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56790334 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚755139950469632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚673386398682112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚701462908069888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9962206963200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚755139974409728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701462233704448 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157683004 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10449571302912 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224572495 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172925000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚691500701106688 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172924998 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100892670 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9924879188190 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚701462908069888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚765589703045120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172925000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚9002.739906535 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚691500683103744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397497495 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚701462233704448 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224572495 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56790334 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚755139950469632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚673386398682112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚701462908069888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9962206963200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚755139974409728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701462233704448 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157683004 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224590975 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56794914 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚755201625147392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚701520080549888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9962370028544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚683367669913150 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397529991 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224590975 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10450427474944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157695296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157695752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172939014 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9963026404352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚701519406171136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100900838 _step‚977 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172939016 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oveNÒ„ rsize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 397529991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9962370028544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673441319852032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚9003.401413058 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 397509925 % _timestamp‚1.7884373186959844e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚701520080549888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9962370028544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚683367669913150 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397529991 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224590975 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10450427474944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157695296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157695752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172939014 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9963026404352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚701519406171136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100900838 _step‚977 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172939016 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 397529991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9962370028544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673441319852032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚9003.401413058 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 397509925 % _timestamp‚1.7884373186959844e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414Ñd&w2›*- _wandbruntime‚9003  _runtime‚90032ø®ÎXœ*Ò 0loss/FRt1/assignment_loss‚1.2872630700469017 /loss/FRt2/assignment_loss‚2.485327964791885 0loss/FRt3/assignment_loss‚2.6835413712721605 .loss/FRt3/detection_loss‚2.533402991600526 /loss/FBt2/detection_loss‚2.5090848803520203 /loss/FBt4/detection_loss‚2.7466148018836973 0loss/SRqqt1/detection_loss‚0.953035282915917 % _timestamp‚1.7884373186982296e+09 _runtime‚9003.402897967 %loss/total_loss‚40.64744678687469 .loss/FRt2/detection_loss‚2.269983751842609 /loss/FRt4/assignment_loss‚2.637633305042982 1loss/FBt1/assignment_loss‚0.09715752541732077 1loss/FBt4/assignment_loss‚0.12126707136631013 /loss/FRt1/detection_loss‚1.7879130605608224 .loss/FRt4/detection_loss‚3.131401961669326 2loss/SRqqt1/assignment_loss‚1.1733582835050598 2loss/SRqqt4/assignment_loss‚1.3700015908107162 0loss/SRqqt2/detection_loss‚1.904152857322319 1loss/SRqqt3/detection_loss‚2.6747277807444334 .loss/FBt3/detection_loss‚2.913104498386383 1loss/SRqqt3/assignment_loss‚1.563641453627497 0loss/SRqqt4/detection_loss‚1.773836970037105 .loss/FBt1/detection_loss‚0.617611305663983 epoch‚9 _step‚978 trainer/global_step‚11899 1loss/SRqqt2/assignment_loss‚1.114687566937623 0loss/FBt2/assignment_loss‚0.2379334277531598 2loss/FBt3/assignment_loss‚0.060764013323932886jIˆ?]*× 0loss/FRt1/assignment_loss‚1.2872630700469017 /loss/FRt2/assignment_loss‚2.485327964791885 0loss/FRt3/assignment_loss‚2.6835413712721605 .loss/FRt3/detection_loss‚2.533402991600526 /loss/FBt2/detection_loss‚2.5090848803520203 /loss/FBt4/detection_loss‚2.7466148018836973 0loss/SRqqt1/detection_loss‚0.953035282915917 % _timestamp‚1.7884373186982296e+09 _runtime‚9003.402897967 %loss/total_loss‚40.64744678687469 .loss/FRt2/detection_loss‚2.269983751842609 /loss/FRt4/assignment_loss‚2.637633305042982 1loss/FBt1/assignment_loss‚0.09715752541732077 1loss/FBt4/assignment_loss‚0.12126707136631013 /loss/FRt1/detection_loss‚1.7879130605608224 .loss/FRt4/detection_loss‚3.131401961669326 2loss/SRqqt1/assignment_loss‚1.1733582835050598 2loss/SRqqt4/assignment_loss‚1.3700015908107162 0loss/SRqqt2/detection_loss‚1.904152857322319 1loss/SRqqt3/detection_loss‚2.6747277807444334 .loss/FBt3/detection_loss‚2.913104498386383 1loss/SRqqt3/assignment_loss‚1.563641453627497 0loss/SRqqt4/detection_loss‚1.773836970037105 .loss/FBt1/detection_loss‚0.617611305663983 epoch‚9 _step‚978 trainer/global_step‚11899 1loss/SRqqt2/assignment_loss‚1.114687566937623 0loss/FBt2/assignment_loss‚0.2379334277531598 2loss/FBt3/assignment_loss‚0.060764013323932886ÒÝéºûž*‚(:’ ÒÆåÔ±„ý gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚88.126' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.625200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºœµ›,Ÿ*‚(:¡ ÒÆåÔ¹áÊ network.recv‚ 3.076521e+06) disk./.usagePercent‚73.87571424053992 disk.sda.in‚0.5" disk.nvme0c0n1.in‚ 1081.0078125 disk.drbd1777.in‚0 network.sent‚ 2.89059e+07 disk.sda1.in‚0 cpu‚25.34414598585942 disk.dm-0.in‚ 1081.0078125 disk.sda.out‚ 366.1015625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 366.1015625! proc.memory.rssMB‚ 2795.578125$ disk./.usageGB‚1100.801528930664# disk.dm-0.out‚53741.45458984375& disk.nvme0n1.out‚53741.45458984375 disk.rbd1.out‚ 4.599609375 disk.sda3.in‚0.5 disk.rbd3.in‚0' proc.memory.availableMB‚ 1027.890625 disk.dm-1.in‚0.5 disk.sda2.out‚0 disk.rbd2.out‚ 236.359375 disk.dm-1.out‚ 366.1015625 disk.rbd1.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0$ memory_percent‚93.72625350952148 disk.sda1.out‚0( disk.nvme0c0n1.out‚53741.45458984375 disk.nvme0n1.in‚ 1081.0078125 disk.rbd0.out‚0 disk.rbd2.in‚0* proc.memory.percent‚17.062854766845703 proc.cpu.threads‚170/6Q“ *‚(:ˆ áÆåÔËÆÑ gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rÉ<¿6¡*‚(:« áÆåÔÁ¯‘ network.sent‚ 2.9014612e+07% disk./.usageGB‚1100.8020248413086 disk.sda3.in‚0.5 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 367.2109375 disk.rbd0.in‚0 disk.dm-0.in‚ 1083.57421875 disk.rbd1.in‚0 disk.rbd1.out‚ 4.599609375( disk.nvme0c0n1.out‚53843.50244140625 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚93.72541904449463 disk.rbd3.in‚0 disk.sda.in‚0.5 disk.dm-1.in‚0.5# disk.nvme0c0n1.in‚ 1083.57421875) proc.memory.availableMB‚ 1028.02734375! proc.memory.rssMB‚ 2795.578125 network.recv‚ 3.083115e+06 disk.sda3.out‚ 367.2109375 disk.sda.out‚ 367.2109375! disk.nvme0n1.in‚ 1083.57421875 proc.cpu.threads‚17 disk.rbd3.out‚0) disk./.usagePercent‚73.87574752151937 disk.drbd1777.in‚0 disk.sda1.in‚0 cpu‚25.343654164521805 disk.rbd2.in‚0 disk.rbd2.out‚ 236.4453125& disk.nvme0n1.out‚53843.50244140625* proc.memory.percent‚17.062854766845703 disk.rbd0.out‚0# disk.dm-0.out‚53843.50244140625 ×»Þ2¢*- _wandbruntime‚9038  _runtime‚9038–iSÈ£* _step‚979 & lr-AdamW/pg1‚9.388235871135617e-08 & lr-AdamW/pg2‚9.388235871135617e-08 trainer/global_step‚11949 % _timestamp‚1.7884373535432684e+09 _runtime‚9038.248004614w;Íͤ*Ç & lr-AdamW/pg1‚9.388235871135617e-08 & lr-AdamW/pg2‚9.388235871135617e-08 trainer/global_step‚11949 % _timestamp‚1.7884373535432684e+09 _runtime‚9038.248004614 _step‚979ÓöÑÄ2¥*- _wandbruntime‚9038  _runtime‚9038Û~μð'¦*êO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚768715127225344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9965714527598 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚686097991214826 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225478509 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173625800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚704320871452672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚704321545818112 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57017090 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10002536406528 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158318158 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10492805593600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9966368965354 trainer/global_step‚11949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚704321545818112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚768715316886528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚686097319737710 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚676131622249472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 399104307 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225498547 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101301070 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173625800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚758222511292928 % _timestamp‚1.7884373535439844e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10003192769024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173625798 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚694318353049088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚704320871452672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225498547 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚676131605210112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚694318353049088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173625798 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10003192769024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚694318335046144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚758222487352832 _runtime‚9038.248933725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57017544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10002536406528 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101301068 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 399124347¶¬Vᶧ*ïO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚704320871452672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚704321545818112 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57017090 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10002536406528 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158318158 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10492805593600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9966368965354 trainer/global_step‚11949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚704321545818112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚768715316886528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚686097319737710 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚676131622249472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 399104307 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225498547 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101301070 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173625800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚758222511292928 % _timestamp‚1.7884373535439844e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10003192769024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173625798 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚694318353049088 UADeviceStatsMonitor.oníÒå¼?_train_batch_start/allocated_bytes.all.freed‚704320871452672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225498547 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚676131605210112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚694318353049088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173625798 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10003192769024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚694318335046144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚758222487352832 _runtime‚9038.248933725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57017544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10002536406528 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101301068 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 399124347 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚768715127225344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9965714527598 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚686097991214826 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225478509 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173625800Ôtâº2¨*- _wandbruntime‚9038  _runtime‚9038éP«ú&©*ôM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚676186526380032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚768777642828800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚686153040303234 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101309236 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10493666251776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9967185400526 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚981 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 399156843 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 173639816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚676186509340672 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225496963 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 173639814 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚9038.910307433 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚704378043919360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚694374688084992 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10004012210176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚694374688084992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 173639814 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚694374706087936 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 399156843 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225517027 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57021614 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 399136777 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10493504738304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158331295 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9966530962562 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225517027 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10003355834368 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10003355834368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158330850 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚686153711780558 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928þŽÙ$ÿ&ª*ùM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225517027 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10003355834368 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10003355834368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158330850 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚686153711780558 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚676186526380032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚768777642828800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚686153040303234 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101309236 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10493666251776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9967185400526 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚981 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 399156843 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 173639816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚676186509340672 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225496963 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 173639814 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚9038.910307433 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚704378043919360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚694374688084992 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10004012210176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚694374688084992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 173639814 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚694374706087936 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 399156843 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225517027 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57021614 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 399136777 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10493504738304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158331295 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9966530962562 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560Õ•ð2«*- _wandbruntime‚9038  _runtime‚9038û’9Ï]¬*× /loss/FRt1/assignment_loss‚1.273476642370224 /loss/FBt3/detection_loss‚3.3664640188217163 /loss/FBt4/detection_loss‚2.4183072835206985 epoch‚9 trainer/global_step‚11949 0loss/SRqqt3/detection_loss‚2.297626423835754 1loss/SRqqt4/detection_loss‚1.8886274693811194 /loss/FBt1/detection_loss‚0.5070259617828903 %loss/total_loss‚41.98928260220359 _step‚982 0loss/FRt4/assignment_loss‚3.7191167332624135 1loss/SRqqt2/assignment_loss‚1.257581102310634 /loss/FRt1/detection_loss‚1.5825497941672801 /loss/FRt2/detection_loss‚2.3094114887973536 1loss/SRqqt2/detection_loss‚1.6465182885771892 .loss/FRt3/assignment_loss‚2.71772586107254 2loss/SRqqt1/assignment_loss‚0.7217413951456547 2loss/SRqqt4/assignment_loss‚1.3144265270613609 1loss/FBt1/assignment_loss‚0.11733778272808307 .loss/FBt2/detection_loss‚3.974073052406311 % _timestamp‚1.7884373542071502e+09 .loss/FRt2/assignment_loss‚2.25893690145534 2loss/SRqqt3/assignment_loss‚1.8280172475746699 1loss/SRqqt1/detection_loss‚1.2536188541352749 _runtime‚ 9038.91181802 2loss/FBt3/assignment_loss‚0.007345846912357956 1loss/FBt4/assignment_loss‚0.14698925979901106 .loss/FRt3/detection_loss‚2.952864120006561 3loss/FBt2/assignment_loss‚0.0005568244232563302 /loss/FRt4/detection_loss‚2.4289437226558985^b­*Ü /loss/FRt4/detection_loss‚2.4289437226558985 /loss/FRt1/assignment_loss‚1.273476642370224 /loss/FBt3/detection_loss‚3.3664640188217163 /loss/FBt4/detection_loss‚2.4183072835206985 epoch‚9 trainer/global_step‚11949 0loss/SRqqt3/detection_loss‚2.297626423835754 1loss/SRqqt4/detection_loss‚1.8886274693811194 /loss/FBt1/detection_loss‚0.5070259617828903 %loss/total_loss‚41.98928260220359 _step‚982 0loss/FRt4/assignment_loss‚3.7191167332624135 1loss/SRqqt2/assignment_loss‚1.257581102310634 /loss/FRt1/detection_loss‚1.5825497941672801 /loss/FRt2/detection_loss‚2.3094114887973536 1loss/SRqqt2/detection_loss‚1.6465182885771892 .loss/FRt3/assignment_loss‚2.71772586107254 2loss/SRqqt1/assignment_loss‚0.7217413951456547 2loss/SRqqt4/assignment_loss‚1.3144265270613609 1loss/FBt1/assignment_loss‚0.11733778272808307 .loss/FBt2/detection_loss‚3.974073052406311 % _timestamp‚1.7884373542071502e+09 .loss/FRt2/assignment_loss‚2.25893690145534 2loss/SRqqt3/assignment_loss‚1.8280172475746699 1loss/SRqqt1/detection_loss‚1.2536188541352749 _runtime‚ 9038.91181802 2loss/FBt3/assignment_loss‚0.007345846912357956 1loss/FBt4/assignment_loss‚0.14698925979901106 .loss/FRt3/detection_loss‚2.952864120006561 3loss/FBt2/assignment_loss‚0.0005568244232563302Öøš‰˜“®*‚(:ˆ ðÆåÔÐäÆŽ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚142.899' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.5798 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 à4Š)¯*‚(:ž ðÆåÔÜ鶑 disk.sda3.out‚ 367.71484375 disk.rbd2.out‚ 236.4609375 disk.sda2.in‚0) disk./.usagePercent‚73.87575724980569 disk.rbd3.in‚0 disk.sda.in‚0.5 disk.dm-1.in‚0.5 disk.rbd1.in‚0! proc.memory.rssMB‚ 2795.578125% disk./.usageGB‚1100.8021697998047 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 367.71484375 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd0.in‚0 network.recv‚ 3.08726e+06 disk.sda3.in‚0.5 disk.dm-0.in‚ 1083.69140625 disk.sda2.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.062854766845703 network.sent‚ 2.9020126e+07! disk.nvme0n1.in‚ 1083.69140625# disk.nvme0c0n1.out‚ 53845.890625) proc.memory.availableMB‚ 1028.23828125 disk.dm-0.out‚ 53845.890625! disk.nvme0n1.out‚ 53845.890625 disk.rbd1.out‚ 4.599609375$ memory_percent‚93.72413158416748 disk.sda.out‚ 367.71484375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1083.69140625 cpu‚25.343140762362417®&çž°*‚(:“ ÿÆåÔŸÐ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.825' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.964999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªùÄ1±*‚(:¦ ÿÆåÔ½öÑ) disk./.usagePercent‚73.87600199300842 disk.dm-0.out‚ 53949.1484375 disk.sda.in‚0.5 disk.sda1.in‚0! disk.nvme0n1.in‚ 1084.57421875 disk.sda2.out‚0 disk.sda3.in‚0.5) proc.memory.availableMB‚ 1027.98828125 disk.rbd0.in‚0 cpu‚25.342661395688857 proc.cpu.threads‚17 network.sent‚ 2.9129162e+07 disk.dm-0.in‚ 1084.57421875 disk.dm-1.in‚0.5 disk.rbd0.out‚0 disk.rbd2.in‚0! disk.rbd2.out‚236.54736328125 disk.rbd3.out‚0" disk.nvme0n1.out‚ 53949.1484375 disk.rbd1.out‚ 4.599609375 disk.sda2.in‚0$ memory_percent‚93.72565746307373! proc.memory.rssMB‚ 2795.578125 disk.dm-1.out‚ 368.37109375$ disk.nvme0c0n1.out‚ 53949.1484375 disk.sda3.out‚ 368.37109375% disk./.usageGB‚1100.8058166503906 network.recv‚ 3.094632e+06 disk.drbd1777.in‚0# disk.nvme0c0n1.in‚ 1084.57421875* proc.memory.percent‚17.062854766845703 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 368.37109375 disk.sda1.out‚0 disk.rbd1.in‚0?ˆ1•2²*- _wandbruntime‚9073  _runtime‚9073ÿ[%Zȳ* % _timestamp‚1.7884373887942638e+09 _runtime‚9073.498982291 _step‚983 & lr-AdamW/pg1‚3.910781469808821e-08 & lr-AdamW/pg2‚3.910781469808821e-08 trainer/global_step‚11999bcïC¾´*Ç _step‚983 & lr-AdamW/pg1‚3.910781469808821e-08 & lr-AdamW/pg2‚3.910781469808821e-08 trainer/global_step‚11999 % _timestamp‚1.7884373887942638e+09 _runtime‚9073ìŽå6.498982291×ÄI[2µ*- _wandbruntime‚9073  _runtime‚9073úT…ïñ'¶*ëO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚761305048176128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226404561 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57245353 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚761305024236032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10043522212352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688884033082234 _runtime‚9073.499837683 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174326598 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚678876828777472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚678876811738112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 226424599 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚697136004991488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10044178574848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688883361605118 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10535691092480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226404561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788437388794966e+09 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101709470 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707179509200896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400751199 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707180183566336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚707179509200896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697136004991488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚771840555898880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158954384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚697135986988544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10043522212352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 trainer/global_step‚11999 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101709468 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400731159 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174326600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚771840739268608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10044178574848 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707179509200896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400751199 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707180183566336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚707179509200896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697136004991488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚771840555898880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158954384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚697135986988544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10043522212352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 trainer/global_step‚11999 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101709468 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400731159 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174326600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚771840739268608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10044178574848 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10043522212352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688884033082234 _runtime‚9073.499837683 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174326598 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚678876828777472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚678876811738112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 226424599 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚697136004991488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10044178574848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688883361605118 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10535691092480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226404561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788437388794966e+09 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101709470 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10044341640192 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10044998016000 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226423015 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10044341640192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10536546710528 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400763629 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚707237356046336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226423015 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 trainer/global_step‚11999 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚9074.165010509 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226443079 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚697192358030336 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174340616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678931732908032 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174340614 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400783695 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174340614 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚985 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688939082170642 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158967074 $ _timestamp‚1.788437389459857e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚771903245624320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400783695 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688939753647966 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10008020739934 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844÷ §zº*úM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚678931715868672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚697192358030336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚707237356046336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10007366301970 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10044341640192 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10044998016000 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226423015 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10044341640192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_s@D€Ä†plit_bytes.small_pool.allocated‚10536546710528 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400763629 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚707237356046336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226423015 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 trainer/global_step‚11999 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚9074.165010509 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226443079 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚697192358030336 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174340616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678931732908032 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174340614 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400783695 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174340614 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚985 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688939082170642 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158967074 $ _timestamp‚1.788437389459857e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚771903245624320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400783695 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688939753647966 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10008020739934 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚697192340027392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚707236681667584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10536374711296 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10044998016000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158967514 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57249438Ùù®Ä2»*- _wandbruntime‚9074  _runtime‚90745ûpá[¼*Õ /loss/FRt3/assignment_loss‚2.748139912262559 1loss/FBt4/assignment_loss‚0.29069597051817586 /loss/FRt2/detection_loss‚2.5870511688684164 epoch‚9 /loss/FRt1/assignment_loss‚1.231469055827783 0loss/FRt4/assignment_loss‚2.4883756962689487 2loss/SRqqt4/assignment_loss‚1.3986924690051148 1loss/SRqqt1/detection_loss‚0.7966462749324433 0loss/SRqqt3/detection_loss‚2.444411108063327 0loss/FRt2/assignment_loss‚2.7902347037666724 2loss/SRqqt1/assignment_loss‚0.8075848667282765 1loss/SRqqt2/assignment_loss‚1.341684431099055 /loss/FRt1/detection_loss‚1.6278431014899089 1loss/SRqqt4/detection_loss‚1.8748828591062472 trainer/global_step‚11999 % _timestamp‚1.7884373894614165e+09 2loss/SRqqt3/assignment_loss‚1.4551339833479788 /loss/FBt1/detection_loss‚0.5018990826848451 ,loss/FBt3/detection_loss‚3.1858069896698 _runtime‚9074.166059526 .loss/FRt4/detection_loss‚2.202596908265918 1loss/FBt1/assignment_loss‚0.11126148884539899 1loss/FBt2/assignment_loss‚0.08541405955786782 /loss/FRt3/detection_loss‚2.7299708798527718 1loss/SRqqt2/detection_loss‚1.7017283086713992 /loss/FBt4/detection_loss‚2.1785995851863516 _step‚986 1loss/FBt3/assignment_loss‚0.06514332680671941 .loss/FBt2/detection_loss‚3.420574426651001 %loss/total_loss‚40.06584065747698\¿ÿv`½*Ú _step‚986 1loss/FBt3/assignment_loss‚0.06514332680671941 .loss/FBt2/detection_loss‚3.420574426651001 %loss/total_loss‚40.06584065747698 /loss/FRt3/assignment_loss‚2.748139912262559 1loss/FBt4/assignment_loss‚0.29069597051817586 /loss/FRt2/detection_loss‚2.5870511688684164 epoch‚9 /loss/FRt1/assignment_loss‚1.231469055827783 0loss/FRt4/assignment_loss‚2.4883756962689487 2loss/SRqqt4/assignment_loss‚1.3986924690051148 1loss/SRqqt1/detection_loss‚0.7966462749324433 0loss/SRqqt3/detection_loss‚2.444411108063327 0loss/FRt2/assignment_loss‚2.7902347037666724 2loss/SRqqt1/assignment_loss‚0.8075848667282765 1loss/SRqqt2/assignment_loss‚1.341684431099055 /loss/FRt1/detection_loss‚1.6278431014899089 1loss/SRqqt4/detection_loss‚1.8748828591062472 trainer/global_step‚11999 % _timestamp‚1.7884373894614165e+09 2loss/SRqqt3/assignment_loss‚1.4551339833479788 /loss/FBt1/detection_loss‚0.5018990826848451 ,loss/FBt3/detection_loss‚3.1858069896698 _runtime‚9074.166059526 .loss/FRt4/detection_loss‚2.202596908265918 1loss/FBt1/assignment_loss‚0.11126148884539899 1loss/FBt2/assignment_loss‚0.08541405955786782 /loss/FRt3/detection_loss‚2.7299708798527718 1loss/SRqqt2/detection_loss‚1.7017283086713992 /loss/FBt4/detection_loss‚2.1785995851863516Ú»è¾*‚(:’ ŽÇåÔ©Ìí gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚87.884' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.576800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾lu<¿*‚(:± ŽÇåÔñÜ…! proc.memory.rssMB‚ 2795.578125) disk./.usagePercent‚73.87601428137006! disk.rbd2.out‚237.93408203125 disk.drbd1777.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚54049.74560546875* proc.memory.percent‚17.062854766845703 network.recv‚ 3.098378e+06% disk./.usageGB‚1100.8059997558594 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0.5 disk.rbd3.out‚0 disk.dm-1.in‚0.5 disk.sda.out‚ 368.93359375! disk.nvme0n1.in‚ 1084.57421875$ memory_percent‚93.72742176055908 disk.rbd1.in‚0 network.sent‚ 2.9134285e+07 disk.sda1.out‚0& disk.nvme0n1.out‚54049.74560546875( disk.nvme0c0n1.out‚54049.74560546875) proc.memory.availableMB‚ 1027.69921875 disk.rbd0.out‚0 disk.sda3.out‚ 368.93359375 disk.sda.in‚0.5 disk.dm-1.out‚ 368.93359375# disk.nvme0c0n1.in‚ 1084.57421875 cpu‚25.34215496353685 disk.rbd1.out‚ 4.599609375 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 1084.57421875 disk.sda1.in‚0 5ƒãÀ*‚(:’ ÇåÔ¯‰æ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚97.692' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.538399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®¼êy1Á*‚(:¦ ÇåÔœ™ë disk.sda2.out‚0 disk.rbd1.out‚ 4.599609375& disk.nvme0n1.out‚54148.90576171875 network.recv‚ 3.105276e+06( disk./.usagePercent‚73.8760288737995 disk.sda.in‚0.5 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.sda.out‚ 369.671875 disk.drbd1777.out‚0 disk.dm-0.in‚ 1085.4140625 disk.rbd1.in‚0! proc.memory.rssMB‚ 2795.578125 network.sent‚ 2.9243231e+07 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.062854766845703% disk./.usageGB‚1100.8062171936035 disk.rbd0.in‚0 disk.sda3.in‚0.5 disk.sda3.out‚ 369.671875 disk.nvme0n1.in‚ 1085.4140625! disk.rbd2.out‚238.51220703125 disk.rbd3.in‚0# disk.dm-0.out‚54148.90576171875 disk.dm-1.in‚0.5 disk.dm-1.out‚ 369.671875( proc.memory.availableMB‚ 1028.1796875 cpu‚25.34164802792564" disk.nvme0c0n1.in‚ 1085.4140625( disk.nvme0c0n1.out‚54148.90576171875$ memory_percent‚93.72448921203613i€€Â*‚(:’ ¬ÇåÔÌ቎ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.88' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.975999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§deA8Ã*‚(:­ ¬ÇåÔÀåÀ disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚93.72456073760986) proc.memory.availableMB‚ 1028.16796875 network.sent‚ 2.9249204e+07 disk.rbd3.in‚0 disk.sda.out‚ 370.25390625 network.recv‚ 3.109823e+06) disk./.usagePercent‚73.87602375364882! disk.rbd2.out‚238.51611328125 disk.drbd1777.out‚0! proc.memory.rssMB‚ 2795.578125 disk.rbd0.in‚0# disk.dm-0.out‚54249.09423828125 disk.sda1.in‚0 disk.dm-1.out‚ 370.25390625 disk.rbd2.in‚0 disk.sda3.in‚0.5 disk.dm-0.in‚ 1085.4140625 disk.sda1.out‚0 disk.dm-1.in‚0.5 disk.nvme0n1.in‚ 1085.4140625& disk.nvme0n1.out‚54249.09423828125 disk.rbd1.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.062854766845703 disk.rbd0.out‚0 disk.sda3.out‚ 370.25390625" disk.nvme0c0n1.in‚ 1085.4140625 cpu‚25.3411924317381 proc.cpu.threads‚17( disk.nvme0c0n1.out‚54249.09423828125% disk./.usageGB‚1100.8061408996582 disk.drbd1777.in‚0 disk.sda.in‚0.5 disk.rbd1.out‚ 4.599609375n(Rú2Ä*- _wandbruntime‚9109  _runtime‚9109Ü)ßXÆÅ*À % _timestamp‚1.7884374243097644e+09 _runtime‚9109.014511861 _step‚987 % lr-AdamW/pg1‚7.93904729941352e-09 % lr-AdamW/pg2‚7.93904729941352e-09 trainer/global_step‚12049mmÉËÆ*Å % lr-AdamW/pg1‚7.93904729941352e-09 % lr-AdamW/pg2‚7.93904729941352e-09 trainer/global_step‚12049 % _timestamp‚1.7884374243097644e+09 _runtime‚9109.014511861 _step‚987Û_Ë°2Ç*- _wandbruntime‚9109  _runtime‚9109v:S—ò'È*ìO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10084508018176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚9109.015426355 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚710038821314560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚699953656933888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚764387561119232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚681622018266112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159591080 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10085164380672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159590619 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102117868 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57473210 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 % _timestamp‚1.7884374243105178e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175027400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚699953656933888 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402378051 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227350651 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10048039644170 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710038146949120 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175027400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175027398 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10578683419136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10047385206414 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚681622035305472 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚691669403472526 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102117870 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚774966244538368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710038146949120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872¹1âIºÉ*ñO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175027398 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10578683419136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10047385206414 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚681622035305472 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚691669403472526 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102117870 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚774966244538368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710038146949120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10084508018176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚9109.015426355 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚710038821314560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚699953656933888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚764387561119232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚681622018266112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159591080 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10085164380672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159590619 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102117868 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57473210 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 % _timestamp‚1.7884374243105178e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175027400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚699953656933888 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402378051 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227350651 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10048039644170 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710038146949120 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175027400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038ܘ«‡Î2Ê*- _wandbruntime‚9109  _runtime‚9109Ò£àû&Ë*õM L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159603376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227349067 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10579539084800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10048201641378 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175041414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10085327446016 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102126038 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159603825 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710095993794560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚764449235796992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚12049 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175041416 $ _timestamp‚1.788437424979811e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚9109.685092507 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10579704792576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚691725795515374 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175041416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚700009991969792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚775028750941696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402410547 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10085983821824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚700009991969792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10048856079342 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 402390481 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402390481 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227369131 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710095319415808 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10085327446016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10085983821824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710095993794560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚681676939436032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57477340 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚764449211856896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚700010009972736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360Úï'Ì*úM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10048201641378 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175041414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10085327446016 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102126038 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159603825 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710095993794560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚764449235796992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚12049 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175041416 $ _timestamp‚1.788437424979811e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚9109.685092507 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10579704792576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚691725795515374 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175041416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚700009991969792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚775028750941696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402410547 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10085983821824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚700009991969792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10048856079342 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 402390481 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402390481 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227369131 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710095319415808 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10085327446016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10085983821824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710095993794560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚681676939436032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57477340 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚764449211856896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚700010009972736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159603376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227349067 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10579539084800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2ÝcjíÔ2Í*- _wandbruntime‚9109  _runtime‚9109€øãCVÎ*Ð .loss/FRt4/detection_loss‚2.447637555996577 trainer/global_step‚12049 $ _timestamp‚1.788437424981841e+09 %loss/total_loss‚40.54182247133952 2loss/SRqqt1/assignment_loss‚0.8520784548018128 1loss/FBt4/assignment_loss‚0.44155707815662026 1loss/SRqqt1/detection_loss‚1.2525088522272805 /loss/FRt3/assignment_loss‚2.617857075506641 2loss/SRqqt3/assignment_loss‚1.6724476601396288 0loss/FBt2/assignment_loss‚0.4649304604002585 _runtime‚9109.686474592 _step‚990 1loss/FBt1/assignment_loss‚0.04216251375421425 1loss/SRqqt3/detection_loss‚2.1515526158469065 /loss/FBt1/detection_loss‚0.6928290881097634 1loss/SRqqt4/detection_loss‚1.8588038167128196 .loss/FBt4/detection_loss‚2.443271528929472 0loss/FRt1/assignment_loss‚1.3881796556849813 0loss/FBt3/assignment_loss‚0.2492752657737583 .loss/FRt1/detection_loss‚2.052486613176243 1loss/SRqqt2/detection_loss‚1.8917807082335154 -loss/FBt3/detection_loss‚2.06078377366066 /loss/FRt4/assignment_loss‚2.908394176430172 2loss/SRqqt2/assignment_loss‚1.2896977910565006 .loss/FRt3/detection_loss‚2.882660085155118 .loss/FRt2/detection_loss‚2.365467722745652 /loss/FRt2/assignment_loss‚2.867639645617059 1loss/SRqqt4/assignment_loss‚1.614790313357774 /loss/FBt2/detection_loss‚2.0330300198660956 epoch‚9îëwl[Ï*Õ /loss/FRt4/assignment_loss‚2.908394176430172 2loss/SRqqt2/assignment_loss‚1.2896977910565006 .loss/FRt3/detection_loss‚2.882660085155118 .loss/FRt2/detection_loss‚2.365467722745652 /loss/FRt2/assignment_loss‚2.867639645617059 1loss/SRqqt4/assignment_loss‚1.614790313357774 /loss/FBt2/detection_loss‚2.0330300198660956 epoch‚9 .loss/FRt4/detection_loss‚2.447637555996577 trainer/global_step‚12049 $ _timestamp‚1.788437424981841e+09 %loss/total_loss‚40.54182247133952 2loss/SRqqt1/assignment_loss‚0.8520784548018128 1loss/FBt4/assignment_loss‚0.44155707815662026 1loss/SRqqt1/detection_loss‚1.2525088522272805 /loss/FRt3/assignment_loss‚2.617857075506641 2loss/SRqqt3/assignment_loss‚1.6724476601396288 0loss/FBt2/assignment_loss‚0.4649304604002585 _runtime‚9109.686474592 _step‚990 1loss/FBt1/assignment_loss‚0.04216251375421425 1loss/SRqqt3/detection_loss‚2.1515526158469065 /loss/FBt1/detection_loss‚0.6928290881097634 1loss/SRqqt4/detection_loss‚1.8588038167128196 .loss/FBt4/detection_loss‚2.443271528929472 0loss/FRt1/assignment_loss‚1.3881796556849813 0loss/FBt3/assignment_loss‚0.2492752657737583 .loss/FRt1/detection_loss‚2.052486613176243 1loss/SRqqt2/detection_loss‚1.8917807082335154 -loss/FBt3/detection_loss‚2.06078377366066Þ Ö“üÐ*‚(:’ »ÇåÔÎðì gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.15' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.630000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n&59Ñ*‚(:® »ÇåÔÏš» disk.sda2.out‚0 disk.sda.out‚ 370.90234375 disk.sda1.out‚0 disk.dm-1.in‚0.5 disk.nvme0n1.in‚ 1085.4140625 disk.rbd1.out‚ 4.599609375 cpu‚25.340715109069656 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 370.90234375( proc.memory.availableMB‚ 1027.9453125 disk.sda3.in‚0.5 disk.sda3.out‚ 370.90234375! disk.rbd2.out‚247.72802734375 disk.drbd1777.out‚0* proc.memory.percent‚17.062854766845703 disk.rbd2.in‚0 disk.rbd3.out‚0 network.recv‚ 3.114337e+06) disk./.usagePercent‚73.87602631372417% disk./.usageGB‚1100.8061790466309" disk.nvme0c0n1.in‚ 1085.4140625( disk.nvme0c0n1.out‚54249.43798828125 disk.drbd1777.in‚0$ memory_percent‚93.72591972351074! proc.memory.rssMB‚ 2795.578125 network.sent‚ 2.9255434e+07 disk.rbd0.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚ 1085.4140625# disk.dm-0.out‚54249.43798828125 disk.sda.in‚0.5& disk.nvme0n1.out‚54249.43798828125à3€HžÒ*‚(:“ ÊÇåÔ‘†Ž gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.733' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.546599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{øM¿-Ó*‚(:¢ ÊÇåÔÈÔ‚ network.recv‚ 3.120472e+06) disk./.usagePercent‚73.87602042555088% disk./.usageGB‚1100.8060913085938! disk.dm-0.out‚54349.009765625 disk.sda1.out‚0 disk.dm-1.out‚ 371.328125 disk.nvme0n1.in‚ 1085.4140625 disk.rbd1.in‚0 disk.sda3.out‚ 371.328125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0! proc.memory.rssMB‚ 2795.578125 cpu‚25.340241646720816 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.drbd1777.out‚0& disk.nvme0c0n1.out‚54349.009765625 disk.sda3.in‚0.5 disk.rbd3.out‚0 network.sent‚ 2.9363186e+07 disk.sda.out‚ 371.328125! disk.rbd2.out‚247.72802734375$ memory_percent‚93.72620582580566( proc.memory.availableMB‚ 1027.8984375 disk.sda.in‚0.5$ disk.nvme0n1.out‚54349.009765625 disk.rbd3.in‚0 disk.dm-0.in‚ 1085.4140625 disk.dm-1.in‚0.5 disk.rbd1.out‚ 4.599609375 disk.sda2.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.062854766845703 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 1085.4140625œ>H’Ô*‚(:‡ ÙÇåÔ±Óê gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚143.506' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.7012 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌÝ £.Õ*‚(:£ ÙÇåÔðÀ‚( proc.memory.availableMB‚ 949.94140625 cpu‚25.34587008032489 proc.cpu.threads‚17 disk.rbd2.out‚ 247.833984375 disk.drbd1777.in‚0 disk.dm-1.out‚ 372.1015625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 1085.4140625 disk.sda2.out‚0# proc.memory.rssMB‚ 2873.16015625 network.sent‚ 2.9369532e+07 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.in‚0.5 disk.sda1.out‚0$ disk.nvme0n1.out‚54448.064453125 disk.rbd2.in‚0! disk.dm-0.out‚54448.064453125 disk.dm-1.in‚0.5& disk.nvme0c0n1.out‚54448.064453125* proc.memory.percent‚17.536377906799316( disk./.usagePercent‚73.8760347619728 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.in‚0.5 disk.dm-0.in‚ 1085.4140625 disk.rbd1.in‚0$ memory_percent‚94.20201778411865 network.recv‚ 3.125056e+06 disk.rbd0.out‚0 disk.sda3.out‚ 372.1015625 disk.rbd1.out‚ 4.748046875% disk./.usageGB‚1100.8063049316406 disk.sda.out‚ 372.1015625 disk.nvme0n1.in‚ 1085.4140625ºÚ[z’Ö*‚(:‡ èÇåÔ‡ò‘Ž gpu.0.gpu‚56.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚87.897' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.5794 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Iߥ.×*‚(:£ èÇåÔËÄô disk.sda.in‚0.5 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 1085.4140625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd2.out‚ 247.833984375 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚54547.4873046875 cpu‚25.353732638498446) disk./.usagePercent‚73.87605114645498 disk.sda2.out‚0 disk.drbd1777.out‚0" disk.dm-0.out‚54547.4873046875 disk.sda1.out‚0$ memory_percent‚94.20080184936523 network.sent‚ 2.9374944e+07 disk.sda3.out‚ 372.578125 disk.sda.out‚ 372.578125 disk.dm-1.in‚0.5' disk.nvme0c0n1.out‚54547.4873046875# proc.memory.rssMB‚ 2873.16015625 disk.rbd1.out‚ 4.748046875 network.recv‚ 3.129327e+06 disk.rbd0.in‚0 disk.sda3.in‚0.5 disk.nvme0n1.in‚ 1085.4140625 disk.drbd1777.in‚0 disk.dm-0.in‚ 1085.4140625 disk.dm-1.out‚ 372.578125& proc.memory.availableMB‚ 950.140625* proc.memory.percent‚17.536377906799316% disk./.usageGB‚1100.8065490722656ù‰¯çœØ*‚(:‘ ÷ÇåÔîéÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.871' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.774199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>MÖ±Ù*‚(:š ÷ÇåÔ©·˜‘ network.sent‚ 2.9381202e+07 disk.sda1.in‚0 disk.dm-1.in‚0.5 disk.rbd1.out‚ 4.780273›8¦ ¦4375" disk.nvme0c0n1.in‚ 1085.4140625( proc.memory.availableMB‚ 949.66796875) disk./.usagePercent‚73.87604013813102 disk.sda1.out‚0 disk.nvme0n1.in‚ 1085.4140625 disk.rbd3.in‚0 disk.dm-0.out‚ 54548.09375 cpu‚25.361138654187172 disk.sda.in‚0.5 disk.dm-1.out‚ 373.2890625 disk.rbd0.in‚0 disk.rbd3.out‚0$ memory_percent‚94.20368671417236 disk.nvme0n1.out‚ 54548.09375 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 1085.4140625 proc.cpu.threads‚17 network.recv‚ 3.133987e+06 disk.sda2.in‚0# proc.memory.rssMB‚ 2873.16015625 disk.sda.out‚ 373.2890625 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 247.896484375 disk.sda3.in‚0.5 disk.sda3.out‚ 373.2890625% disk./.usageGB‚1100.8063850402832" disk.nvme0c0n1.out‚ 54548.09375 disk.drbd1777.out‚0* proc.memory.percent‚17.536377906799316+™3W2Ú*- _wandbruntime‚9186  _runtime‚9186ÇãgæX*Û*ÒT ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚10136182841856 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚704068802937856 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚779403236743168 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 404597957 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚57882315 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚714204967776768 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚10136182841856 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚685602481833984 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 404581978 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚714205478609920 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 176131548 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚10099354775594 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 404581978 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 404597957 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 176131550 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _runtime‚9186.178072238 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚152 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 176131550 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚695701345695186 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 228466407 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚10136675672064 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 _step‚991 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚714204967776768 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚704068784934912 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚695701853648938 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚57882467 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚10098863861202 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 228450430 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚10632199781888 trainer/global_step‚12089 $ _timestamp‚1.788437501472918e+09 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚704068802937856 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 160653767 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚768771152305152 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 176131548 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 404597957 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 176131550 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _runtime‚9186.178072238 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚152 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 176131550 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚695701345695186 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 228466407 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚10136675672064 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 _step‚991 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚714204967776768 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚704068784934912 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚695701853648938 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚57882467 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚10098863861202 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 228450430 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚10632199781888 trainer/global_step‚12089 $ _timestamp‚1.788437501472918e+09 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚704068802937856 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 160653767 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚768771152305152 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430ßS9_2Ý*- _wandbruntime‚9187  _runtime‚9187ˆˆï'Þ*ÝR MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚154 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚714242162932224 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 176142356 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚57886852 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 228457732 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 160664808 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚714242673765376 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚714242162932224 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 102777958 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 _step‚992 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 102777956 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚704105632966656 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚152 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 228473709 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚704105614963712 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚57887004 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚10099718385002 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884375023714712e+09 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 404600086 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 404600086 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 176142356 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 228473709 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚10137040798720 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115343872 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚704105614963712 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚768809051978752 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚685637775593472 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚10137040798720 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚9187.076535427 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚10632449Ö¦ýL564672 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752*ˆ›h)ß*âR RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115343872 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚704105614963712 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚768809051978752 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚685637775593472 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚10137040798720 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚9187.076535427 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚10632449564672 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚154 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚714242162932224 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 176142356 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚57886852 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 228457732 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 160664808 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚714242673765376 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚714242162932224 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 102777958 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 _step‚992 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 102777956 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚704105632966656 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚152 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 228473709 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚704105614963712 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚57887004 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚10099718385002 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884375023714712e+09 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 404600086 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 404600086 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 176142356 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 228473709 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚10137040798720 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0à`$jà*‚(:’ †ÈåÔÁ¦ß gpu.0.gpu‚9.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚164.561' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.912200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,KÄ…/á*‚(:¤ †ÈåÔ¼›ü disk.nvme0n1.in‚ 1085.4140625 disk.sda3.out‚ 373.734375 network.sent‚ 2.9385583e+07 disk.sda2.out‚0 disk.rbd0.in‚0 cpu‚25.368442082435397 disk.dm-0.in‚ 1085.4140625" disk.nvme0c0n1.in‚ 1085.4140625 disk.sda3.in‚0.5$ memory_percent‚94.25647258758545 disk.drbd1777.in‚0" disk.dm-0.out‚54647.7255859375 disk.sda1.out‚0 disk.rbd1.out‚ 4.7802734375' disk.nvme0c0n1.out‚54647.7255859375 disk.sda2.in‚0* proc.memory.percent‚17.591571807861328 network.recv‚ 3.136928e+06 disk.drbd1777.out‚0 disk.sda.out‚ 373.734375 disk.sda1.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 941.01953125% disk./.usageGB‚1100.8063659667969 disk.rbd3.out‚0 disk.dm-1.in‚0.5 disk.dm-1.out‚ 373.734375) disk./.usagePercent‚73.87603885809334% disk.nvme0n1.out‚54647.7255859375 disk.rbd2.in‚0 disk.rbd2.out‚ 247.900390625! proc.memory.rssMB‚ 2882.203125 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda.in‚0.5 disk.rbd0.out‚0œI„î2â*- _wandbruntime‚9199  _runtime‚9199ùFØ$Nã*ͦ *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3962889611721039 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5430629849433899 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5335540175437927 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005281417164951563 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16907352209091187 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8484849333763123 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333432674408 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 particle/accuracy_5_of_7‚1 particle/accuracy_7_of_7‚1 trainer/global_step‚12089 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23403875529766083 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.28172391653060913 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983899593353271 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.991355299949646 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.24256758391857147 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 _step‚993 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9702637195587158 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07317603379487991 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23096774518489838 particle/accuracy_4_of_4‚1 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.190476194024086 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5261526703834534 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015086237341165543 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9645865559577942 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 particle/accuracy_1_of_7‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5274050235748291 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998635470867157 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.45136454701423645 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_6‚0.47435900568962097 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12938007712364197 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13659249246120453 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021757816895842552 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5442163944244385 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5199999809265137 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3263888955116272 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5170301198959351 *jet/accuracy_2_of_5‚0.4543779194355011 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4545685350894928 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5186058282852173 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08704935759305954 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9768145084381104 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5687775015830994 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2169611006975174 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14750957489013672 jet/accuracy_7_of_7‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5230526328086853 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5821429491043091 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 jet/accuracy_6_of_7‚0 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0826149508357048 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010976340621709824 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 +jet/accuracy_4_of_6‚0.03846153989434242 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3240740895271301 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9870635867118835 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.44346410036087036 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5358348488807678 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48964717984199524 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,jet/accuracy_4_of_5‚0.012096773833036423 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5213368535041809 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.49044379591941833 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5036177635192871 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1821623593568802 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13991768658161163 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3263888955116272 jet/accuracy_6_of_6‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10680510848760605 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 *jet/accuracy_1_of_4‚0.9304083585739136 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 epoch‚9 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5445402264595032 +jet/accuracy_4_of_4‚0.00627895025536418 particle/accuracy_1_of_2‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895178079605103 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5892088413238525 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18402138352394104 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5599340796470642 particle/accuracy_1_of_1‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17535163462162018 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.990576446056366 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04511008784174919 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24292927980422974 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3916666805744171 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.18421052396297455 particle/accuracy_2_of_6‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5815151333808899 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4344105124473572 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5499550700187683 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1812036633491516 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 jet/accuracy_4_of_7‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.030221719294786453 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4122768044471741 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21387435495853424 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4965914487838745 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,particle/sensitivity‚0.31798434257507324 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5602945685386658 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2072046995162964 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.539758026599884 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 +jet/accuracy_3_of_4‚0.07659120857715607 *jet/accuracy_3_of_5‚0.1031746119260788 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333432674408 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5838016867637634 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 particle/accuracy_3_of_3‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03675703704357147 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3234051465988159 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9913591146469116 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5590448379516602 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16124926507472992 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19852308928966522 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43052488565444946 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0026455027982592583 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5397295951843262 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 +jet/accuracy_2_of_4‚0.39214226603507996 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.641188383102417 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 particle/accuracy_6_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.552629828453064 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5393415093421936 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.297311931848526 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.45891791582107544 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17971539497375488 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2697077989578247 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3648809492588043 particle/accuracy_6_of_6‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.989993155002594 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04556478187441826 (particle/f_score‚0.40022706985473633 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5026524662971497 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5118244290351868 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28101176023483276 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4814814329147339 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16804936528205872 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.25831490755081177 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28054964542388916 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27595677971839905 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4583333730697632 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 jet/accuracy_5_of_6‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 6 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.185484379529953 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0.5 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.54365074634552 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9899256825447083 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4382951557636261 7 Purity/"_›À0E1FRt2SRqqt0FBt/FRt_purity‚0.1576695740222931 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_1_of_6‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878621697425842 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 jet/accuracy_1_of_7‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4074074327945709 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6111111044883728 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15623128414154053 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39791664481163025 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2926587164402008 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19396552443504333 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6100693941116333 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.33712121844291687 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.602809727191925 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.030008913949131966 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.582635223865509 particle/accuracy_1_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611342906952 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19513744115829468 particle/accuracy_5_of_5‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13777777552604675 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17506003379821777 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5301176905632019 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4313720166683197 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_2_of_2‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.19230769574642181 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.259929895401001 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2370370775461197 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27804455161094666 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9919330477714539 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.026486380025744438 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9208037853240967 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09078811854124069 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49152302742004395 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8156464099884033 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 $ _timestamp‚1.788437515097642e+09 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18751996755599976 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08353787660598755 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.45511990785598755 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8789648413658142 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166567325592 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _runtime‚9199.804970719 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5273700952529907 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5457717776298523 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9680315852165222 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.36739975214004517 (particle/accuracy‚0.8513247966766357 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5240658521652222 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027965938672423363 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7069699764251709 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6075549721717834 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9500098824501038 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.10606061667203903 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2374221831560135 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322917461395264 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.42491579055786133 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3487223982810974 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5447530746459961 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9925737977027893 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25736191868782043 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014808230102062225 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19730618596076965 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20601853728294373 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5335540175437927 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17339180409908295 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.519407331943512 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4199555814266205 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.41576576232910156 *jet/accuracy_1_of_1‚0.5014861226081848 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5356385707855225 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1702127605676651 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.314605712890625 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5589504837989807 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5057011842727661 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17876257002353668 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4011628031730652 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46462956070899963 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5542132258415222 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6754385828971863 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9986394643783569 *jet/accuracy_1_of_2‚0.7720981240272522 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4470306932926178 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4866666793823242 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10842239856719971 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8484849333763123 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0037735849618911743 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14709937572479248 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5625878572463989 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9855169057846069 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16933727264404297 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18402138352394104 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25431033968925476 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5428768992424011 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21001480519771576 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_2_of_2‚0.11371628940105438 jet/accuracy_3_of_7‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 particle/accuracy_5_of_6‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2056403011083603 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08898808807134628 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3795658051967621 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.37773367762565613 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5027639269828796 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18280763924121857 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19170764088630676 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 5validation_average_jet_accuracy‚0.448900431394577 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.203125 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23224468529224396 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37035420536994934 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9850148558616638 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 'loss/val_total_loss‚41.460609258745 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6012812256813049 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16320733726024628Ò)êÂ:ä*Ò¦ ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17506003379821777 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5301176905632019 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4313720166683197 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_2_of_2‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.19230769574642181 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.259929895401001 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2370370775461197 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27804455161094666 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9919330477714539 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.026486380025744438 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9208037853240967 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09078811854124069 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49152302742004395 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8156464099884033 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 $ _timestamp‚1.788437515097642e+09 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18751996755599976 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08353787660598755 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.45511990785598755 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8789648413658142 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166567325592 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _runtime‚9199.804970719 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5273700952529907 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5457717776298523 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9680315852165222 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.36739975214004517 (particle/accuracy‚0.8513247966766357 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5240658521652222 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027965938672423363 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7069699764251709 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6075549721717834 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9500098824501038 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.10606061667203903 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2374221831560135 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322917461395264 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.42491579055786133 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3487223982810974 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5447530746459961 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9925737977027893 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25736191868782043 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014808230102062225 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19730618596076965 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20601853728294373 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5335540175437927 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17339180409908295 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.519407331943512 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4199555814266205 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.41576576232910156 *jet/accuracy_1_of_1‚0.5014861226081848 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5356385707855225 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1702127605676651 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.314605712890625 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5589504837989807 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5057011842727661 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17876257002353668 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4011628031730652 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46462956070899963 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5542132258415222 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6754385828971863 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9986394643783569 *jet/accuracy_1_of_2‚0.7720981240272522 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4470306932926178 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4866666793823242 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10842239856719971 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8484849333763123 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0037735849618911743 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14709937572479248 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5625878572463989 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9855169057846069 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16933727264404297 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18402138352394104 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25431033968925476 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5428768992424011 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21001480519771576 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_2_of_2‚0.11371628940105438 jet/accuracy_3_of_7‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9918614625930786 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 particle/accuracy_5_of_6‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2056403011083603 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08898808807134628 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974507093429565 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3795658051967621 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.37773367762565613 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5027639269828796 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18280763924121857 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19170764088630676 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 5validation_average_jet_accuracy‚0.448900431394577 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.203125 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23224468529224396 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37035420536994934 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9850148558616638 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 'loss/val_total_loss‚41.460609258745 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6012812256813049 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16320733726024628 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.39628896117210e&þÚ—X39 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5430629849433899 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5335540175437927 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005281417164951563 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16907352209091187 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8484849333763123 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333432674408 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 particle/accuracy_5_of_7‚1 particle/accuracy_7_of_7‚1 trainer/global_step‚12089 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23403875529766083 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.28172391653060913 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983899593353271 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.991355299949646 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.24256758391857147 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 _step‚993 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9702637195587158 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07317603379487991 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23096774518489838 particle/accuracy_4_of_4‚1 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.190476194024086 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5261526703834534 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015086237341165543 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9645865559577942 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 particle/accuracy_1_of_7‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5274050235748291 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998635470867157 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.45136454701423645 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_6‚0.47435900568962097 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12938007712364197 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13659249246120453 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021757816895842552 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5442163944244385 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5199999809265137 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3263888955116272 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5170301198959351 *jet/accuracy_2_of_5‚0.4543779194355011 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4545685350894928 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5186058282852173 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08704935759305954 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9768145084381104 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5687775015830994 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2169611006975174 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14750957489013672 jet/accuracy_7_of_7‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5230526328086853 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5821429491043091 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 jet/accuracy_6_of_7‚0 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0826149508357048 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010976340621709824 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 +jet/accuracy_4_of_6‚0.03846153989434242 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3240740895271301 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9870635867118835 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.44346410036087036 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5358348488807678 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48964717984199524 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,jet/accuracy_4_of_5‚0.012096773833036423 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5213368535041809 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.49044379591941833 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5036177635192871 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1821623593568802 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13991768658161163 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3263888955116272 jet/accuracy_6_of_6‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10680510848760605 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 *jet/accuracy_1_of_4‚0.9304083585739136 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 epoch‚9 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5445402264595032 +jet/accuracy_4_of_4‚0.00627895025536418 particle/accuracy_1_of_2‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895178079605103 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5892088413238525 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18402138352394104 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5599340796470642 particle/accuracy_1_of_1‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17535163462162018 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.990576446056366 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04511008784174919 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24292927980422974 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3916666805744171 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.18421052396297455 particle/accuracy_2_of_6‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5815151333808899 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4344105124473572 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5499550700187683 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1812036633491516 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 jet/accuracy_4_of_7‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.030221719294786453 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4122768044471741 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21387435495853424 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4965914487838745 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,particle/sensitivity‚0.31798434257507324 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5602945685386658 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2072046995162964 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.539758026599884 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 +jet/accuracy_3_of_4‚0.07659120857715607 *jet/accuracy_3_of_5‚0.1031746119260788 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333432674408 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5838016867637634 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 particle/accuracy_3_of_3‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03675703704357147 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3234051465988159 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9913591146469116 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5590448379516602 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16124926507472992 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19852308928966522 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43052488565444946 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0026455027982592583 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5397295951843262 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 +jet/accuracy_2_of_4‚0.39214226603507996 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.641188383102417 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 particle/accuracy_6_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.552629828453064 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5393415093421936 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.297311931848526 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.45891791582107544 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17971539497375488 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2697077989578247 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3648809492588043 particle/accuracy_6_of_6‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.989993155002594 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04556478187441826 (particle/f_score‚0.40022706985473633 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5026524662971497 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5118244290351868 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28101176023483276 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4814814329147339 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16804936528205872 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.25831490755081177 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28054964542388916 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27595677971839905 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4583333730697632 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 jet/accuracy_5_of_6‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 6 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.185484379529953 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0.5 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.54365074634552 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9899256825447083 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4382951557636261 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1576695740222931 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_1_of_6‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878621697425842 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 jet/accuracy_1_of_7‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4074074327945709 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6111111044883728 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15623128414154053 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39791664481163025 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2926587164402008 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19396552443504333 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6100693941116333 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.33712121844291687 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.602809727191925 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.030008913949131966 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.582635223865509 particle/accuracy_1_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611342906952 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19513744115829468 particle/accuracy_5_of_5‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13777777552604675 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1áy?åVRå* c3mvx01yj@ ÈåÔÈòðï0`Trainer.fit` stopped: `max_epochs=10` reached. “ª@ëŸæ* c3mvx01yjŒ ÈåÔ˜™Ë‡zEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:14:30 • 0:00:00 1.40it/s v_num: x01yn·%ç* c3mvx01yj ÈåÔÐîà‡ í3V62è*- _wandbruntime‚9206  _runtime‚9206Þ½%-é*‚(’öG